Rob's Notes 3: Reducing Thrash in "Cannot Fail" Software Projects
“Thrash” is an excessive amount of friction that negatively impacts teams trying to advance a project or initiative. Some of it is avoidable.
tldr; Two big causes of team thrash are retaining optionality (often via deferring decisions or avoiding conflicts) and asking multiple people to answer the same question/solve the same problem. Here are some ideas to counter this and other problems in these challenging projects.
“Cannot Fail” software projects (CFs) are wide-ranging initiatives which are technically complex and have significant business consequences for companies. CFs can be very challenging and they exist for companies large and small alike, though the tips here are slightly more relevant to large technology organizations. CFs have many hyphens: they are deadline-bound high-impact cross-company technically-constrained projects driven by a mix of (sometimes conflicting and shifting) external and internal requirements.
Note: a company’s specific incentives and culture often build in a certain amount of thrash by default. Because these values and expectations may be ingrained, they need to be countered thoughtfully. Also, this advice is by no means comprehensive, but I hope some of these suggestions and quotes may be of service to you and the teams you lead, in avoiding thrash:
Project Structure
Create “single threaded owners” (STOs). Create clearly differentiated workstreams and assign people to lead them that are clearly known so everyone can direct their questions to that central point of contact. Amazon calls these types of folks STOs and I’ve heard Apple refers to them as Directly Responsible Individuals (DRI). If you have someone working for you on one of these cannot-fail projects, do as much as you can to take other responsibilities off their plate
Create forcing functions based on need. Regular executive reviews are a good forcing function for the team to have its story straight; but collapse and consolidate them wherever possible to reduce repetitive work. Make sure these reviews have the (small group of) correct attendees and solid agendas to make good use of everyone’s time. Don’t let these meetings balloon in size; share notes with the people who would have attended the prior reviews (before you consolidated them!) so they get the necessary context. Once leadership is confident the team is on a good path these should change to less frequent discussions, or simple email updates.
Manage the number of cooks in the kitchen. Many people at the company see or hear how important these CFs are, and want to help solve them, but be wary of adding unnecessary swirl by piling people on (especially ‘leaders’ vs ‘do-ers’). Don’t ask the team for ad hoc updates if there are reasonable regular ones already available. Don’t request an in-person briefing to get up to speed if written documents can provide reasonable context already. The team should be sure to keep pointing people back to those if they exist, and should update them whenever possible.
Communications
There is no such thing as over-communication. No matter how many times you think you’ve said something, the n-th time will be someone’s first time truly hearing it and internalizing it. Establish a rhythm and cadence for communication proactively so that stakeholders understand what they’ll hear and when. Be open to feedback so that the communication remains useful. This is especially true when discussing and accepting risk (technical, business, legal or otherwise) associated with pursuing a particular direction. One caveat is be careful communicating still-pending decisions to the team (see below).
Restate goals and timelines frequently. Log important decisions made and to-be-made and include them in your comms updates. Aim for frequent clear restatement of project goals, plans, and expected timeline - not just deltas - which will feel redundant but is not. It may force disagreements to bubble up sooner. Build mutual knowledge about the current state of knowns and unknowns while avoiding having myriad stakeholders deluge the team with multiple “TPS Report requests”.
Be plainspoken without dumbing down the complexity. Execs will rarely be able to understand the level of nuance that the subject matter expert may know exists under the hood. Yes, you reading this might be just that expert! Use simple, concise and clear language that explains why a particular idea might be complex or difficult to implement, and be descriptive about the consequences of that challenge, e.g. real world impact on customers, the business, degradation in reliability, stability, etc.
Also: look inward and help your management help you. Communicate up the chain of command and take responsibility for informing your management chain about progress. I have previously quoted Jocko Willink’s excellent book Extreme Ownership: "If your boss isn’t making a decision in a timely manner, or providing necessary support for you and you team, don’t blame the boss – first blame yourself. Examine what you can do to better convey the critical information for decisions to be made and support allocated."
Decisions and constraints
Get specific about what decisions you need. Be direct and unequivocal about when a decision must be made (for engineering feasibility, compliance or otherwise). Maintaining an open door when timelines are too short exacerbates any existing risks, non-linearly.
Prioritize how you ask for feedback from leaders. Leadership input on desired outcomes is one important input but not the main or only driver. There may be incredible technical complexity that limits how helpful execs can be in giving feedback without ramping on an area. Work to move stakeholders in or out: push them to get more or less involved depending if they are experts and/or have the bandwidth to understand the issues deeply. This may require execs deciding to divide responsibilities and focus on one workstream and not another!
Set objection deadlines and explain your default decisions. Be specific about what feedback you need and when you need it by. Is this an FYI communication but there’s some openness to considering an objection? When does that expire? Effective FYI communication of key decisions can further empower decision making at the lowest level. Does the team need a tie break because the team can’t agree on the best option among a final two in consideration? Be clear about what happens once your deadline passes, e.g. “unless we hear objections by MM/DD, the plan of record is XX” or “if we don’t receive a tiebreak here, we will consider the following escalation path for in person review…”
Conflict resolution is a deliverable. Ensure you’re making “consistent, frequent progress on convergence of disagreements - with deadlines.” Create a fast escalation path and make sure teams know it exists and how to access it. Establish whether conflicts are real or soft and push back on 'soft no’s' (e.g. 'can we run these queries/experiments') when the new info would not change the decision one way or the other.
Be clear on how decisions are made and by whom. Workstream leaders need to always ensure they track the minimally viable set of people needed for a final 'yes' decision on any issue. This may change during the course of a project and this requires management. Decisions should be made at the lowest possible level based on impact. This is enabled by having a sense what one-way doors look like for your product or business, and by consulting affected teams. Give them clear windows and mechanisms to escalate pending decisions.
Recognize the difference between optional and mandatory work. CFs are mandatory by definition. That mandate is an attractive vehicle for people to add additional work under that scope that isn't strictly required. Be judicious in adding marginal work under the banner of an CF - there are good things that come from “not wasting a crisis” to get certain needed things prioritized but make sure to look under the hood at the proposed details.
Sustainability and scalability
Ask for help. Need a second set of eyes on an escalation email to ensure you are being objective? Ask someone who has experience effectively escalating to review your draft. You will think that you can’t slow down and ask for help, but forcing yourself to slow down allows you to take a step back, incorporate others’ perspectives and solidify mentors and sponsors along the way. Make it easy for others to help you by maintaining a ‘captain’s log’ and/or a quick summary of the situation so that, as necessary, someone can cover for you when you need a break.
Manage unavoidable changes carefully with teams. Workstream leads should be aware of potential changes coming up, but should not thrash their teams with possibilities (since these disruptions will spread out recursively down and sideways in ripples!) until a change is actually decided on or near certain.
Speed outweighs finish level. Things may start out messy and that’s okay. Lower your expectation of the polish level on the CF project communications in exchange for learning about issues and disagreements sooner. It’s not just the formatting: sometimes junior folks may “hide” bad news thinking they can fix it themselves. Don’t do this. Make it okay for anyone on the team to ask for help early and often. Get even less hung up on stuff that doesn’t matter (if you are a VP who doesn’t like the name the team came up with for an internal-only metric, your time might be better spent elsewhere). Easier said than done, but since multiple teams will need to collaborate across org lines to get things done, you’ll need to lower your regular level of defensiveness or territoriality which could be challenging depending on your corporate culture.
Finally - remember that while you and your colleagues are all on the same team, you also owe each other direct and timely feedback, and accepting this without getting defensive may be hard! This is really important both in terms of asking your leadership for help as well as providing them the feedback that they’re thrashing you and/or the team. Your organization may not be used to this kind of directness, so embark on this with care. Always recognize most people have good intent and just want to help; and that giving feedback is not about avoiding accountability.
Appendix: Words of wisdom from practitioners:
Recently I asked several product, engineering and policy leaders I trust (with experience at various tech companies) for their advice working on these CFs. My main prompt to them was advice for execs on how to reduce thrash for these kinds of projects, and here is some of what they said:
“Make a decision sooner instead of keeping all options open.”
“Don’t ask multiple people to do the same work- this happens when execs each use their chain of command to try to inform on an issue. Get an exec leader and coordinate info gathering and response”
“The best take away for me was to implement some sort of log for ‘key decisions’. There was so much thrash by just having to relitigate old decisions. Also trusting the team to make decisions as opposed to execs thinking they are in the best position to. How many times have we heard them say “Why can’t you just do X?””
“Things may not feel thrashy to you but the further down the chain people get the more thrashy it feels. Ensure good communication not just from yourself but from your direct reports to their teams. Encourage escalation and not making it a bad thing when teams disagree. It's common for teams to have conflicting goals that they can't resolve on their own. The important thing is helping them to quickly recognize those and uplevel quickly. Then it's important for the senior managers to resolve quickly.”
“I would also add do your homework as senior exec and align your decision with your peers and stakeholders. It is very thrashy to push that down.”
“Things and needs can change rapidly… be diligent on how likely something is needed before communicating it with the team - or keep a small group of people who you consult on how things might change, but don't broadcast it to everyone. That said, be willing to tell your team when you don't know yet what will need to happen and/or if you expect things to be fluid. Sometimes I found more communication even if it was me telling them I didn't have answers yet, was better than nothing. Oh also, respect time zones!”