Recreating Engineering Excellence as a Leader

Recreating Engineering Excellence as a Leader

Note to my non-techie audience: This article is a bit more technical than usual, but as always, there’s a leadership lesson.

This week, I’m reflecting on my time as an engineer and am considering what that means for my leadership practice. There are things that I did back then, that I now oversee. The technologies and times have changed, but a lot of the principles have not.

20 years ago, I worked as a senior systems engineer for a financial services company. The team I worked on was called “the middleware team,” which is an outdated term nowadays and was even esoteric back then. It was an architectural term. Client-server architectures were on their way out, and newer web-based multi-tier applications were in. Middleware was the term we applied to the tech stack between the database and the end-user web browser. We ran various application servers, web servers, load balancers, identity services, and message queuing technologies.

Organizationally, the team sat between the application developers and the infrastructure platform teams. Modern tech teams call this function DevOps Engineering or Site Reliability Engineering. From my viewpoint, we covered both disciplines. As a team, we managed hundreds of applications on thousands of servers.

There’s a mindset that was driven into me back then, that I’ve never shaken. I credit my leaders back then for drilling it into me and my co-workers for maintaining high standards.

Here are the lessons I learned as a systems engineer:

  1. Security was on me. I wouldn’t let a developer get away with an insecure configuration. I didn’t care if it ran that way on his dev box. I wasn’t going to let it go to test or production without meeting our stringent security standards. Sure, developers were pushy, but so was I.
  2. I documented everything. Everything had a change control. Everything had a backout plan. If I got hit by a bus, one of my co-workers could have read my documentation and recreated the application environment perfectly if they needed to.
  3. Availability was paramount. I learned a very hard lesson early in my career. You can read about it here. From then on, I insisted that all mission-critical applications had both high availability and disaster recovery. I also insisted that these capabilities were validated regularly. I forced myself to get comfortable with multi-region failover, so exercising it would be easy.
  4. I monitored my apps. I wanted to know about availability issues and performance problems before the end users called. I was on-call and responsible anyway, and it was always better to find out about issues from the monitors than the humans.

That’s a nice trip down memory lane. So what?

Here’s the challenge. This overall function is still in my purview. Nearly all of the technologies and techniques have changed. Instead of data centers, we have clouds. Instead of traditional monitoring, we have observability. Instead of manually configured application servers, we have Infrastructure as Code tools. Instead of manually documented application installations, we have CI/CD pipelines.

Yet the high-level outcomes of security, auditability, availability, and performance still matter.

I’m not an engineer anymore. I’m not qualified to run any of the technologies in our current environment (except for perhaps a few legacy apps).

I’m big on these principles still. That hasn’t changed. I strongly resist the urge to utter the words “Back in my day, this is how we did it…” because I know how that would have sounded to me when I was an engineer.

Here’s my challenge:

I must recreate the conditions for my team to flourish. The actions were invisible to me at the time, but some leaders several layers above me 20 years ago, created the conditions for me to thrive as a systems engineer. This is what they did, and therefore, this is what I must do:

  1. Psychological safety was paramount. It was safe to experiment, safe to innovate, safe to fail, and safe to succeed. I didn’t know it at the time, but that wasn’t accidental. It was intentional and difficult to maintain.
  2. I had as much training as I needed. If I wanted to take a class or pursue a certification, the answer was always yes. Someone high above me defended that budget line.
  3. I had a great manager (Hi Jeff!). He cared, pushed, and protected his team. The leaders above him developed him. I must develop the leaders and managers on my team to be their very best.
  4. The expectations for excellence were high, not just from management but also from my peers, upstream developers, and downstream platform teams. Everyone pursued excellence and it was contagious.

You might be thinking, wow Zach, that company culture you described sounds amazing. Where was it? It was at a mortgage company, called GMAC-RFC. It didn’t survive the mortgage crisis of 2008, but it did create a diaspora of tech leaders who know what “good” looks like. I wrote more about that here and here.

So, as a tech leader, not an engineer, can I create a culture that replicates engineering excellence in a completely different context, with completely different technologies and people, 20 years later?

That’s the goal. Certain leadership theories evolve because we learn more about what works and what doesn’t. Other leadership principles are timeless. I do my best to pay forward the good experiences I learned early on and evolve the practice of technology leadership for the modern times we are in today. If I do that well, we’re unstoppable.

At CHS, we’re blessed with a top-notch team. We’re already great, but we’re still getting greater. One of my leaders (Hi Lauren!) has the famous Vince Lombardi quote in her email signature: “Perfection is not attainable, but if we chase perfection, we can catch excellence.”

That’s how I see it. It’s true, I’m never satisfied. I’m careful not to wear people out, but I continue to drive my team to greater and greater levels of proficiency. As an engineer and as a leader, that’s how I see it.

I hope you’ve found my reflection helpful. Engineering and leadership are two very different disciplines. Evolving my thinking from one discipline to the next has taken me decades, and the journey continues.

Comments are closed.