In almost every class that I teach or organization that I coach I get the following question: “What metrics should we use to determine if a team is performing well?” I can tell from the context of the question that most people are asking about the development team role, not the full Scrum team (composed of a product owner, ScrumMaster, and development team) and not the organization as a whole.
I agree there are times that I want to know how the development team is performing, so I will address that question in this blog. However, that question is of much less relevance to me than questions like: “How are we performing as an agile organization?” or “How is a Scrum team performing?” Answers to these questions provide insight at a more economically relevant and actionable level.
At the organizational level I want to know if are we delivering desirable customer value in an efficient manner. To understand if we are, I would measure things like:
- Idle work versus idle workers — measure when and how often the flow of work is being impeded (rather than how busy we keep people).
- Delivery of working and validated assets — delivering on time and on budget doesn’t really matter if we don’t deliver a product people want to use.
- How quickly we as an organization are learning — using things like innovation accounting (a Lean Startup concept) that evaluates how fast we are learning as a measure of our progress towards converging on a business-valuable result.
Below the organizational level I want to focus on the full Scrum team. It takes all three roles to deliver the right customer value and to deliver it well. So I am interested in measuring things like "Is the Scrum team delivering good value every sprint?" (more on this shortly).
But what if we want to measure just the development team (hereinafter just “team”)? Perhaps we want to know whether we made good choices when assigning people to teams. If development teams are going to be long-lived assets, then we will want to know how well these teams are performing.
I begin the discussion of measuring development teams by ruling out the use of velocity and instead suggest other measures including:
- Team meets its sprint goal most every time
- Product owner believes that the team is delivering good economic value
- Team is working at a sustainable pace
- Team has or is acquiring relevant T-shaped skills
- Rate at which the team is learning
Velocity (Do Not Use This)
For many people, velocity is an obvious way to measure team performance. Unfortunately, velocity should only be used as a planning tool (e.g., release planning) and as a team diagnostic measure (e.g., are process improvements working). It should not be used as a performance metric in an attempt to judge productivity. Velocity is simply too easily gamed to use it for productivity measurement purposes. If team members know that velocity is the measure that people will use to judge the performance of their team, they can arbitrarily increase the size of the product backlog items (point inflation) or cut corners to get more things done each sprint. In my Essential Scrum book, I provide a detailed description of what velocity is, how it should be used, and how it can be misused. For our purposes here, I will not include velocity in a multidimensional measure of team performance.
Achieving Sprint Goals
Okay velocity is out, so what should we use to measure team performance? One indicator of how a team is doing is the frequency at which the team meets its sprint goal. As a rule, every sprint should have a goal (like an executive summary statement) that the team commits to achieving. I want a team to achieve its goal most every sprint. I’d prefer that a team not achieve its goal every single sprint since that might be an indicator that the team is habitually under-committing. Occasionally, I want a team to have made a solid attempt to achieve the goal but to miss it, for the right reasons (e.g., the goal was bit of a stretch). That better indicates to me that the team is likely performing at its reasonable limit.
An easy and appropriate check and balance on “low-ball” team commitments is to ask the product owner whether or not she is getting good value each sprint. Most sprints are performed at a fixed cost—we know who is on the team and we know the duration of the sprint, therefore we know how much it costs to run the sprint. Let’s say it costs $20k to run each sprint. An important question to ask the product owner is “Do you feel like you are getting at least $20k of value each sprint?” A good product owner will know the answer to this question. If the product owner says, “Yeah, I am happy with the value that team is delivering,” then that is an indication of good team performance.
As a note, the product owner has overall responsibility for the economic outcome at both the sprint and the release level. So, if the product owner naively spends $20k and asks the team to work on product backlog items that deliver $10k value, then the product owner is not performing well. Overall, delivering good value is a responsibility owned by the entire Scrum team (product owner, ScrumMaster, and development team), but it is factor to consider when evaluating the development team.
Working at a Sustainable Pace
I also want to know if the team is delivering good value nearly every sprint at a sustainable pace. We have all seen team members work 80 hours a week to deliver on their sprint goal. The first reaction might be, “See, I have a great performing team because they are willing to work so hard to get the job done!” Every now and then it might be necessary to work longer and harder during a sprint to get the job done. There can always be the unpredictable event that causes a bit of a crunch. However, people can’t work at an unsustainable pace for an extended period of time, so team members that work 80 hours a week won’t be a “great” team for long, they will soon be a "burnt-out" team. So, my third indicator of a good performing team is does it delivery good value most every sprint while working at a sustainable pace.
Increasing T-shaped Skills
Another measure of good team performance is: "Do team members have and are they working to further develop their T-shaped skills?" T-shaped skills is a metaphor I use to describe a person with deep vertical skills in a specialized area as well as broad but not necessarily very deep skills in other relevant areas. Teams composed of members with T-shaped skills are more resilient to fluctuations in the work, since it is likely that more than one team member can work in a given area, and therefore teams can swarm people to areas where there is an abundance of work to perform. I believe that the degree to which team members have and are acquiring T-shaped skills is an important indicator of team health and performance.
Learning Fast and Frequently
Finally, there is the rate at which the team is learning. In particular, I am interested in the rate at which the team is completing its learning loop of: assume, build, feedback, inspect, and adapt. High-performing teams never let important assumptions live long before validating them and acting on what they learned. High-performing teams organize the work in a way that maximizes their ability to learn important details fast, so they can adapt to what they are learning.
Given the importance of learning fast and frequently, I would apply this measure at the organization and Scrum team levels as well. Just like teams that are quickly learning important information tend to do the best work and outperform slow-learning teams, organizations that learn the quickest tend to trounce their competition.
To summarize, when measuring performance, first consider higher levels like organizational and Scrum team levels. And, when evaluating the development team, don’t use velocity. Instead consider a multidimensional set of measures like the ones I discussed in this blog to get a more encompassing picture of team performance.
One final observation. In my experience good managers within an organization don’t really need any formal metrics to determine how well their teams are performing. Such managers stay in close contact with the people doing the work and they are keen observers of what is actually taking place. Ask a good manager about the performance of any team she is associated with, including its strengths and weaknesses, and she will be able to give you an immediate, informed answer to that question.