I want to start, by apologizing for the recent outages of the email service. And now I would like to try to explain what has happened, and what we are doing to prevent this from happening ever again.
To begin with my email server is rented from a vendor, it is in a server farm with many other similar large servers, and my space is shared with other resellers on this larger server that have similar setups as mine. This larger server is huge, and has thousands of resellers all with their own customers email accounts on it. My vendor is one of the only providers of email service in the world that allows people like me to resell email hosting service in this way. Almost all email hosting providers charge from $5 to $7 per email inbox. My vendor allows me to give you as many email accounts as you want under your domain for a flat fee that is far less than the other providers offer. This does not even take into account the fact that big tech reads every email you send or receive and indexes it so they can advertise to you. So for these reasons and many more I have chosen this vendor, and for about the last 4 years we have nothing but spotless service from them.
And that brings us to the recent outages. From the very beginning of this first crash, I have been in close communication with the owner of the email hosting company, and he has been explaining all of the technical details of what has been happening all along the way. To say that the string of events that has lead us to this moment, is bad luck is a total understatement. But I can assure you that none of what has happened thus far was the result of negligence or mismanagement.
The first outage (last week) was the result of storage array failing. The storage array is where all of the individual email files are stored for your email accounts. These kind of things happen from time to time, and that is why we have backups. Several attempts were made to restore the storage array, and unfortunately it was so badly damaged that the server could not be resurrected. Thus a new server was spun-up, and the backed up files were restored back to if. Unfortunately the backup service was off-site, and the restore took a lot of time to complete.
Once we were operational again last week, a plan was put into place to create a secondary server to mirror the primary server, that would continuously be kept up-to-date, so that if something similar happened in the future that we could switch over to the mirror server and be back up and running in a couple minutes. The majority of last week was devoted to setting up this mirror server, and tweaking minor issues that popped up along the way.
That brings us to Saturday morning. The plan was to spend the weekend, getting everything synced between the live server and the mirror. But the storage array failed again, probably due to all of the massive amounts of data being written and read from it. The storage array consists of multiple drives that each hold parts of the other drives in the array, and it is designed to be able to withstand a failure of a single drive. The idea being that if a drive fails you swap it out with a new drive and the array is rebuilt from the data on the other drives. On Saturday morning a drive failed, and during the procedure to rebuild the array with the new drive, a second drive failed. This was catastrophic to the array because it meant big chunks of data would now be lost, and the whole array could not be trusted, anymore.
So a new array has been built and we are now in the midst of another restore from the slow off-site backup service. I don’t know how long it will be before the email is back online and you can get your messages again, and I can’t be sure if we will not have duplicate old emails come into your inbox again. We are putting the importance of not losing email over the possibility that you might see duplicates or deleted old mail. So we will have to see what happens after the restore completes.
So now we have to wait until the array is restored, and the email is flowing again, but after that, we have to decide what we are going to do next. I have been asked by a couple of my customers what I am planning to do to help mitigate this situation from happening again. The short answers is that I am not planning on moving away from this vendor. The owner and I have a good relationship, and I can reach out to him personally with anything I need. That is not something you can say about any other email service provider in the world. I am also not willing to have big tech use all of my (and your) email to profile and advertise to us. And lastly, during the long span of time I have been with this vendor, I have had nothing but stellar service and support. This outage, while currently very disrupting, has been minor compared to the long solid long history of service that they have provided up to this point.
That being said, I understand how important email is to our businesses and our personal lives. And so if you would like to move your service to another provider, I completely understand…, and I will help in any way I can to get you moved as quickly as possible.
But I truly believe that this vendor is the one that I want to stay with, and I believe that lessons have been learned, and the likelihood of this set of circumstances happening again will be very unlikely.
I’m truly sorry for the inconvenience, and I hope you will accept my sincere apology. Please let me know how you would like to proceed and if you would like to discuss anything further. I can be reached by text at 928-699-8308, or at my backup email address: jesse.waitz@gmail.com
Thanks again for your patience, and your loyalty over the years. I hope that I can earn your trust back in the coming years. Please let me know if there is anything else I can do for you.
Sincerely,
Jesse Waitz.