Apology email from Leadpages about internet outage
This is a great apology email from LeadPages about a 4 hour downtime on people’s pages. We love how honest they are in this email, and they step-by-step explain WHAT WENT WRONG, and HOW THEY WILL SOLVE IT in the future.
Subject: We’re sorry
Body copy:
Hi,
I’m Steve Brain, Leadpages’ Chief Technology Officer.
On Tuesday, many of our customers experienced an outage of their published pages. We know how important your landing pages are to your business and we deeply apologize for the impact of this outage. Now that we’re back online, we want to share exactly what happened, why it happened, and—most importantly—how we will prevent it from happening again.
To make a long story short, due to a technical error, the third party that manages our domain name registrations mistakenly shut down our publishing domain. As soon as our team identified this as the source of the problem, we contacted the third party and restored all affected pages. Total downtime was approximately 4 hours.
Now, let’s dive into the nitty-gritty of what went down:
Q: WHAT CAUSED THE OUTAGE?
A: Our 3rd party domain registrar (NameCheap) reacted to a phishing report and shut down the DNS for the lpages.co publishing domain. A significant portion of our pages are published on lpages.co.
Q: WHY DID IT TAKE SO LONG TO RESOLVE?
A: Unfortunately, NameCheap has no phone support and required us to submit our case by email. It took us about an hour to understand the requirements after multiple chat discussions. Then they had a 2-hour turnaround to re-enable the domain. Typically, DNS propagation takes hours when changes occur, so once the domain was re-enabled customers had varying results based on the DNS servers they were hitting.
Q: WHY DID YOU NOT MOVE TO ANOTHER DOMAIN REGISTRAR TO RESOLVE THE PROBLEM?
A: As part of the abuse handling procedures, NameCheap.com locked the DNS entry for lpages.co, making it impossible for us to transfer it to another registrar.
Q: HOW WILL THIS PROBLEM BE PREVENTED IN THE FUTURE?
A: We are moving our domain registrations to a single service with stronger emergency support to prevent this from happening again.
Q: WHAT CAUSED NAMECHEAP TO THINK LPAGES.CO WAS A PHISHING SITE?
A: There were some reasons that lpages.co could be perceived as suspicious. It hosted the same Leadpages login screen as leadpages.net. Further complicating things, leadpages.net is controlled by a different domain registrar. Therefore, at first glance, it would appear that lpages.co was a phishing site for leadpages.net IDs.
A deeper dive would have revealed that lpages.co was hosting tens of thousands of sites, was a legitimate business, and was active. A simple follow-up with us would have confirmed this and explained the relationship between leadpages.net, lpages.co, and Avenue 81 (the company that owns and operates Leadpages, and who has been paying NameCheap.com as a registrar for years).
We have been in contact with the COO at NameCheap.com and they have admitted they failed on this occasion and are reviewing their internal procedures. They understandably are unwilling to publicize procedural changes for the fear that they will be abused by malicious parties.
Q: HOW COULD WE RECEIVE BETTER UPDATES DURING AN OUTAGE LIKE THIS?
A: Our team is working to improve our communications for events such as this one. During the outage, we provided updates on our status page and also our Facebook site, with the goal of providing updates every 30 minutes. In some cases, we exceeded this, in others we were too slow to provide updates. We recognize customers expect to see frequent and regular updates, even if just to communicate that we are aware of the situation and working to correct it. It is possible to subscribe to email notifications on our status page.
At Leadpages, we strive to be customer-centric in everything we do. We believe our success and your success are one and the same. Still, sometimes we fall short of delivering exceptional service, which is why we deeply apologize for the impact of this outage.
As your partner, we also value transparency. That’s why we thought it important to share what happened, why, and what we’re doing to learn from it, and prevent a recurrence. It’s all part of our commitment to your success in all of your business-building endeavors.
Sincerely,
Steve Brain
Chief Technology Officer at Leadpages
