A Rough Day at the Office for Digital Takumi
Nothing ruins a great end-of-week atmosphere quite like discovering that all the websites and emails you oversee have crashed. This is precisely what happened to Christopher Bradbury and his company, Digital Takumi, last week. AWS ultimately resolved the issue, but Bradbury’s blunders provide a clear lesson on what to avoid.
The Dreadful Discovery
It was on Thursday, July 16, when Bradbury realized that all the websites and Google Workspace emails associated with the domains he manages were down. The websites are hosted through Route 53, AWS’ DNS/hosting service—and none were operational. While sifting through emails, Bradbury discovered multiple AWS notifications in his spam folder, alerting him that his payment card had expired and that the account would be terminated if he didn’t take action. The problem was, he missed it because he hadn’t checked that folder.
Spam Folders and Missing Employees
“AWS had been sending billing notifications, but they had been diverted to spam and, in some instances, sent to a former employee,” Bradbury admitted. “I take responsibility for overlooking those notifications,” he added, but, of course, that didn’t assist his customers’ sites.
MFA Chaos
Things could have been resolved earlier, but Bradbury faced additional problems: “The root account had MFA enabled through a software authenticator on an old laptop, which was out of commission,” Bradbury clarified. Without that authentication code generator, he was locked out. He had been circumventing it using email for MFA—not the smartest choice.
Email Troubles
Then came the kicker: “The AWS recovery process required email verification at the registered root address,” Bradbury mentioned. “However, that email address was tied to one of the domains whose DNS was hosted in the suspended AWS account, preventing me from receiving the verification email.”
The Recovery Process
Bradbury’s subsequent action was to contact AWS using a different email, which yielded no results. Based on a recommendation, he established another AWS account and purchased business support access, but support engineers refused to discuss the other account until ownership was verified.
Endless Transfers
“In the following days, I spoke with numerous AWS teams, including Billing and Account Recovery. Passed around like a package, no one could complete the ownership verification or restore account access,” Bradbury expressed. He was eager to pay AWS, but time dragged on before they could accept his funds.
Finally a Solution
While we were on this subject, after discussions with both Amazon and Bradbury, he reached out to us to say that site access had been restored, he’d logged in, settled the overdue invoices, established a new payment method, reset his MFA keys, and generally resolved all the chaos he had been causing. It’s unfortunate that someone had to go through the AWS billing ordeal as a lesson for others, so let this serve as a cautionary tale for anyone managing client sites through AWS.
Key Takeaways
“First and foremost, settle your AWS bills,” Bradbury noted in an email, emphasizing the importance of ensuring that the recovery email isn’t associated with any domain of the sites you manage through that profile. Additionally, “stop taking shortcuts” with MFA. “This isn’t a vast infrastructure account; we handle a single company marketing website and a few domains through Route 53,” Bradbury explained. “It simply shows that even the smallest AWS setups can be vital to a business, and you must adhere to proper practices and processes.”
Summary: Why Did the AWS Chicken Cross the Road? To Update Its Payment Method!
There you have it, everyone! Even the slightest mistake can cause a major headache. Pay your bills, regularly check your emails, and stop messing around with MFA shortcuts. Not just a tale of misfortune—consider it a lesson from the grand university of tech missteps.
