How to lead large programs like Eddie

I’m often asked for advice about running large programs, so I have summarized the lessons I have learned as a service to others. I hope they help you on your next major program!

Who is this guy? Why do you care? I’m currently Senior Distinguished Engineer for Physical Infrastructure in the Office of the CTO at Equinix. Previously, I was the uber technical lead for data center software at Google, among many other roles. That said, you probably don’t care because this isn’t going to be on a test.

People respect you when you respect them

First and foremost, you need to treat everyone with respect. It’s the right thing to do, and it’s also necessary for you to be reliably successful at scale.

What does it mean to respect people anyway? To me, it means:

• Practice active listening. Take the time to fully internalize feedback before responding to it. You might just learn something, but if not, people will be more likely to offer you more feedback in the future where you do learn something.
• Treat others’ feedback with care. If someone makes a suggestion, you have two choices: defend your position or adapt your solution to account for it. If you choose to defend, politely explain the arguments in favor of your solution and engage in a constructive dialogue about how their suggestion might be improved. If you choose to incorporate it, follow up to demonstrate how their feedback impacted the solution to ensure they keep offering good suggestions.
• Always be constructive. As a senior leader, you should always be moving the ball forward, breaking the problem down so it can be delegated further, and creating a supportive culture. Otherwise, you’re just complaining.

If you respect people, they will give you the benefit of the doubt and defend your ideas when you’re not around. It will create an environment where people want to do their best. People will volunteer for tasks instead of waiting to be assigned. People will feel psychologically safe enough to let you know when you’re making a mistake, rather than sitting back and waiting for you to fail. All of these qualities are essential to successfully executing a major program.

Your job is to solve the problem

Odds are, you are not singularly brilliant. However, you are probably leading a team of really smart people. Your job is to wrangle these people into solving the problem. Usually, that means assembling their individual ideas into a coherent solution by picking the best ideas, adding some sugar on top, and working with everyone to agree on the unified solution.

Solving the problem does not require you to conceive every idea, make every decision, or take all the credit. In fact, these behaviors are actively harmful because they suck the oxygen out of the room. These behaviors demoralize and disempower the team which is ultimately harmful to you because your job is to solve the problem.

Therefore, acknowledge others at every opportunity. They will appreciate the credit and become more committed to the team. Plus, it helps their careers by making them look good in front of their leadership.

Your most important task is clarifying the problem

People usually only understand a problem as they have experienced it. This results in everyone solving a fraction of the problem. For sufficiently large problems, enough people will be involved that their solutions become mutually incompatible, which leads to conflict, wasted effort, and eventually failure.

You can address this by ensuring everyone is solving the same problem. I have had success using some combination of the following techniques:

• Define the problem mathematically. Capacity planning is fairly simple mathematically. Request new capacity (C) to address the shortfall between demand (D) and supply (S): C = D — S, where D and S are random variables. Then, you can get teams to work on the hard problems of predicting D and S.
• Focus the problem. There are a million ways to break a service, but there are fewer ways to break it quickly enough that you can’t stop it from impacting users. You can generalize the problem by clearly describing the things that can break it quickly.
• Consider externally observable behaviors. Many tech leads (TLs) will focus on internal details. To steal a line from Westworld, “If you can’t tell, does it matter?” By focusing on externally observable behaviors, you can eliminate many unnecessary subproblems.

I wish I could create a step-by-step playbook for how to do this. Unfortunately, every problem is different enough that a generalized solution has eluded me.

Be prescriptive on requirements and flexible on implementations

Respecting your team members implies respecting their autonomy and expertise. You can’t tell them what to do and how to do it. However, you do need to ensure their implementation actually solves the problem. You can do this by being prescriptive about the requirements of their solution.

For example, you may require that each service must be sharded by location, rate limit actions, provide turnup automation callable by a centralized service, and maintain 99.9% per-location availability. It’s not your responsibility to ensure that every service uses the latest automation framework because that wasn’t one of your requirements. Each team is responsible for its own service, as toilsome as that may be. Your job is to ensure that the pieces fit together into a solution that solves the problem.

People must act like you when you are not there

You can’t be everywhere at once. You can’t be a decision bottleneck. You can’t work on this problem space forever. Therefore, you need people to fend for themselves. You can do this by training people to make the same decisions as you when you are not in the room.

I adopt two strategies for this:

• Create rules for others to follow. This follows from prescriptive requirements. You need to reinforce these requirements by training the organization and creating accountability structures for the team. For example, you may create an orientation course (record it!), schedule a one-time session for existing employees, and convince your leadership to hold managers accountable for their TLs following the rules.
• Mentor selected leaders. You are probably going to move to a new project when this one is complete, but the TLs will remain. You should take this time to mentor them how to solve their problems and run large programs of their own. Over time, the organization becomes more capable, trusts you more, and reinforces the values you have created.

Manage information overload

Attention is your most scarce resource because you’re going to be bombarded with information constantly. Use email filters to turn down the noise and turn up the signal. I used multiple Gmail inboxes to separate out actionable tasks like code reviews, document comments, and approvals from conversational emails. I unsubscribed from lots of things I used to have time to read. Don’t let that one person cc: the program mailing list on every bug. Find a system that works for you.

Actively manage your task lists and ruthlessly delegate anything that someone else could do to create time for things that only you can do. In cases where you can’t delegate, actively communicate priorities and status then do the most important things. Leadership will come to understand that you are human and that they need to support you with more people or temper their expectations.

Make your executives work for you

Any time you start a new large program, you will experience many executive-induced distractions. They will ask you for reviews and status updates, exhausting your attention economy unless you do something about it.

First, recognize that executives are bothering you because they care about your success. Once you realize this, you can turn the tables by providing them with concrete actions they can do to help you. The hard part is convincing them that those actions are necessary.

You can make executives work for you by creating accountability structures. This allows you to pinpoint the problems and ask them to ensure those organizations take action to address the problems.

• Early in the program, you’re looking for TLs to partner with, so your accountability structure is a list of services that have assigned TLs. You should request that your leadership assist you by providing TLs so you can solve their problem.
• Once you have scoped the problem, you need services to author designs that address your requirements, so your accountability structure is a list of ETAs for each service’s designs. You should be explicit that leadership needs to prioritize this work if they want it faster.
• Once you have designs, you need to track deliverables. Work with your program manager to create dashboards tracking each deliverable and hold each team accountable for marking each task as complete by reporting progress using only the dashboard. Keep your leadership in the loop with a monthly status email or review to let them know which teams are doing well and which are behind schedule. Explicitly acknowledge the teams doing well.
• As you move to sustaining, have a final review with leadership to remind them of managers’ responsibilities. Hold them accountable with automated monitoring and alerting, if possible.

You’ll find that executives will leave you alone as soon as you’ve demonstrated your ability to exercise accountability structures. They’ll either switch to offline email updates or reduce the frequency to a manageable interval.

Would you like to know more?

There are many books on the subject of technical leadership. I found the following particularly useful:

• What do executives do, anyway? Pennarun describes how executives’ jobs are to apply formal processes to enforce values.
• Give and Take: Grant describes why people who help others get ahead of selfish actors. The tl;dr is that selectively supporting others creates a virtuous cycle that helps you in the long run.
• Staff Engineer: Leadership Beyond the Management Track: Larson provides a comprehensive view of senior leader responsibilities based on his experiences and interviewing many other technical leads.
• The Art of Doing Science and Engineering: Learning to Learn: Hamming describes how to approach ambiguity and overcome organizational inertia. I find it interesting to observe how one of the most influential minds in computing worked.
• Good Strategy, Bad Strategy: Leadership is about choice. Rumelt describes strategy as the scientific method in a way that can help you simplify problems for others.
• Trillion Dollar Coach: When you look past the bloviating text, Bill Campbell taught a human-centric leadership style: creating psychological safety, treating people with respect, directly addressing conflict, and creating accountability.