an on-call love story

10 years ago

It’s late August 2014, on an expectedly cool summer evening in San Francisco.  We’re walking back to our place in Russian Hill from a sentimental bar, the Lion Pub, where we hung out after a long week of work.  As we cross over Alta Plaza and the benches that look down on the Golden Gate Bridge, I reach into my buzzing pocket and pull out my phone.  An all too common event in the 3 years that we’d been dating.

I watch the disappointment rush over her face.  How many times had we had an evening ruined by the backpack I was constantly carrying around?  How many times had I left her at dinner and sat in the car, trying to revive whatever platform or service needed help?  How many times that year had HBase been the bane of my existence?  How many sleepless nights had affected the days and plans ahead?  How many pre-cloud days would I have to drive down to a data center in Santa Clara in the middle of the night and run a KVM across uneven floors all to end up sleeping in our colo cage?

We sit down on a bench and I start my process as she stares off into the park.

Yet, unlike all the times before, it’s all a ruse.  I reach into my backpack and instead of pulling out my laptop, I pull out an engagement ring.

A life of on-call

Looking back, it made perfect sense to propose using being on-call as a way to throw her off the scent.  It was a huge part of my life – our lives – and was intimately tied to my work identity.  But if I’m honest, it’s also quite sad.

At this point, I’d been on-call for 4 years straight.  4 years on very small “operations” teams at Klout and Flipboard where I was primary or secondary the entire time.  Add another 10 years before that where I was part of larger teams but still carried the beeper frequently.  Yes, I actually was on-call during the days when you would hand off a beeper once a week.  I would go on to another 9 years of leading teams and organizations that were responsible for platforms that served millions of users.  24 years of being on-call.

It’s taken its toll on me, the weight of unicorns upon your shoulders in the middle of the night.  That level of constant stress (and fear) hasn’t been good for my health and it’s taken years of pushing in an opposite direction to remedy.

But it’s also a love story for sure. I built my career around being calm and collected during the highest of pressure and delivering.  I never wanted to be a hero, but these herculean efforts were rewarded as my career advanced.  But it’s never been easy.  And the ceremony and rituals around being on-call are just that: rituals that have improved things at the edges, but mostly are ceremonial.

That’s why we’re building beeps.  On-call has been the status quo for far too long.  We’re moving away from highly-trained “operations” engineers who have built their careers around this to modern organizations where every software engineer is on-call.  It’s time to re-think on-call from first principles and find a path that not only benefits the reliability of an app or platform, but also helps take care of the folks who make that happen.

New technology should be enabling on-call, not making the systems more complex to understand. This path  makes things easier to orient around a problem and make decisions in those crucial moments.  The next generation need not suffer the ways we did in the past, and it’s my personal mission to fix that.

A better on-call

Lately, as we’re in the earlier stages of building beeps in 2024, on-call for me is waking up when my kids cry in the middle of the night because they see a spider or they’re having a nightmare.  It’s a different kind of tending to, but with systems so complex that there’s no easy fix beyond a hug or two.

But sooner rather than later, I know my phone will beep again while I’m sitting on a bench watching my son play tennis.  This time, I’ll pull out my phone and beeps will have already gathered the signals I need to make an informed decision on what to do next.  And just maybe, I’ll be able to quickly get back to what matters most in life…