As many of you know, our website was down or non-functioning for most of last week, and getting it back up was largely my responsibility, though not solely mine. I worked hard with fellow web team member Cory Westover as well as support people at Amazon Web Services to get the site back up and running.
For those of you who are more technically inclined and savvy with networking, what we suspect happened was a Distributed Denial-of-Service or DDoS attack, in which the server was flooded with an excess of traffic, rendering it mostly unusable. This can be a difficult thing to fight, as the excess traffic often comes from a variety of different IP addresses via proxy servers. For each address you block, there are dozens more these hits are coming from. You can block whole sets of IP addresses at a time, but that runs the risk of also blocking legitimate traffic, which may be worth it in the meantime.
While my only prior credit in the paper itself was for a photo of a burned house, if you’ve seen our website, you’ve seen my work, as much of the current site is my design, and I handle most content updates for it. Many of you have also spoken to me over the phone regarding your subscriptions or even other websites we build, host, and manage, as well as email hosting.
Something I struggled with and was reminded of by multiple people, was that the problem was not my fault. It was something that happened due to circumstances beyond my control in an area that was my responsibility. It’s just easy to see the latter as the former. I was similarly reminded not to stress too much about it, despite the seriousness of the issue. I suppose the reason I did feel like it was my fault was all the work I put into the site before.
In many ways, I felt that if I was unable to resolve the problem, then I was letting everyone down. I suppose what I needed to remember was that it wasn’t a problem wholly unique to the Times site, and I’ve resolved other crises on websites we host in the past. In the three years that I’ve worked at the Times, our web hosting architecture has changed immensely, largely through work of mine. Because of this, resolving performance issues on our servers wasn’t new to me, even if I had more difficulty than I was used to.
I’m sure many of you have had a crisis in your wheelhouse at your jobs before, where it wasn’t your fault but fixing it was your responsibility. This can weigh heavily on you, and make you question your fitness for your job. From my experience last week, I can say that when such a crisis arises, that it’s important to be willing to reach out for help, and to remember all your past successes on the job should outweigh this challenge in your mind. Furthermore, by resolving the crisis, you’ll be better prepared for when something similar happens in the future, or maybe your solution can provide safeguards against such a repeat incident.
What I suppose was weighing on my mind regarding this ordeal was the ways in which responsibility in a crisis can affect one’s mental health. One of the most important things is the willingness to ask for help when you need it. I know it’s not something I’m great at, but I’m working at getting better.
