Estimating.dev

Estimate Confidently

There is no team – Calibrating teams vs individual estimators 

by

Focusing on the ‘team’ as the primary estimator is a mistake in modern agile style software management. 

The typical claim is that “We will estimate these stories and build up a log of velocity data for the team.”, or “Story points are a measure of effort, but only for this particular team”. The problem is that there is no ‘team’. A software engineering team is not a stable long lasting construct. The average engineer stays at a company for less than two years. Your team is going to change month over month. Losing a senior engineer and gaining a new hire is going to effect your ‘velocity’. 

In addition to constantly changing personnel, the work the team does constantly changes. In software you don’t build the same thing over and over. You do greenfield development, launch, add new features, maintain existing software,  rearchitect software, wrangle legacy software and finally deprecate the project. In a software project that lasts 10 years the team spends maybe a year or two in each phase. And the team is doing totally different types of work in each phase. 

So the first problem with focusing on the ‘team’ is that teams don’t really last longer at the company than engineers do. Teams evolve, their personnel constantly change, their work changes and their goals change. 

The second problem with focusing on the ‘team’ is averaging engineers estimates together. 

Averaging estimates robs engineers of the ability to calibrate their estimates. 

The typical software estimate ends up as a single number in a JIRA ticket with no providence. We don’t know exactly where it came from. We know Jim the project manager added that number to the ticket, but at the same time Jim doesn’t actually make estimates. The estimate came from a ‘planning poker’ meeting where estimates from the entire team were combined into one big number. 

The problem is that to make good estimates you need to be calibrated. You have to make estimates for various tasks and then come back later to compare your accuracy. Do you typically estimate low or high? Are you estimating much larger ranges of time than you should be? But if you don’t write down your individual estimates anywhere, you’ll never be able to evaluate your estimates. 

Combining estimates from multiple sources is a key part of coming up with accurate estimates at an organization level. But if the only data you save is an aggregate estimate of the ‘team’, individual engineers can’t calibrate their estimates. 

Consider tracking estimates at the individual level as opposed to the ‘team’ level. Individual engineers have a more stable identity than software teams. You can actually compare engineer Steve at year five in his career with Steve at year nine in his career. Over the same period a software engineering team wouldn’t be recognizable. And Steve can actually review his estimates and calibrate his output vs his predictions.

Agile Estimation Theatre

by

We’ve all experienced what I like to call Agile Estimation Theatre. Imagine it’s the beginning of the sprint and your team is holding a sprint planning meeting. The project manager has built up a set of stories a few of which have subtasks. Now you will spend the next two hours trying to create estimates for each of those stories. 

The project manager starts by reading the first story. Someone asks a question about whether ECS or EC2 are a better solution for the problem which devolves into a 20 minute conversation about the merits of various AWS services. The project manager is forced to stop discussion to leave any possibility of estimating all the stories slated for the sprint. You open your planning poker tab and evaluate your options 1,3,5,7,13,21,34, 55. After a moment you decide on 13 since you are sure the task will take at least a day or two. One of your coworkers then asks whether the numbers are days or hours. That question leads to a 10 minute discussion of what exactly story points are and how they are definitely not a unit of time. The teams votes range from 5 to 34. And then project manager overrides the score to be 21 since that’s what the senior engineer picked. 

The two hours continue to drag on. By the blessed end of the meeting your team has ‘estimated’ 4 out of 20 single sentence stories. Since the sprint starts tomorrow the project manager holds a quick meeting with the senior engineer to estimate the rest of the stories before 5pm. 

The above tale represents what I like to call agile estimation theatre. We have a sprint planning meeting where we’re supposed to involve the entire team in estimation, so we schedule a two hour meeting and hope for the best. Sometimes it actually works, but often we end up with questionable estimates for single sentence tasks. 

The fundamental problem with Agile Estimation Theatre is not investing enough time. To get good estimates you need to have complete stories. You need to break down subtasks and specify them more thoroughly than a single sentence can possibly handle. Then you need to have your team spend time reviewing the stories individually and coming up with estimates. The problem is that it doesn’t make sense to spend several days out of a two week sprint on estimation. To do it right you need to have well specified stories. Building out the stories will take at least a few engineer days. Next you need to have the entire team spend 2 hours each estimating stories individually. Finally you add together the individual estimates and do the sprint planning meeting. Now repeat that process again every two weeks. 

The breakdown is that the expected value of the estimates is typically less than the cost of spending a few days to produce good estimates. The organization intuitively knows this which is why we spend two hours in sprint planning instead of two days. But because we want to do scrum or agile ‘properly’ we need estimates. The end results is that we denature the estimates to comply with the process. 

Estimating Unbounded Agile tasks

by

I have been reading Software Estimation (https://www.amazon.com/Software-Estimation-Demystifying-Developer-Practices/dp/0735605351) after a recent conversation with one of the product owners on my team. The Senior engineer on my team has pushed us into a ‘pointless’ estimating process where the estimate is just the count of the subtasks in the story. 

The Product owner wanted us to provide time or t-shirt size estimates earlier in the process before we did the work to break up the story into subtasks. This discussion broke down into people arguing over agile… 

Personally, I am quite disillusioned with how we create estimates a part of the Software Development lifecycle. The standard  estimation technique in my experience is to compare a new project with a past project based solely on personal memory and gut feel. Then to double your estimate from what ever you came up with. The computer science and engineering classes I took did not cover anything related to estimating projects. Which is a little strange, since other engineering disciplines like Mechanical and Electrical engineering cover estimates. 

The business needs estimates so that we can coordinate work with plans, make deals and hopefully profit. If you promise a client something and the schedule slips 6 months that is a problem. If you sign a 10 million dollar advertising deal and the delivery schedule slips 3 months it is a problem. So the business needs estimates that are accurate to plan things. 

But if we are just guessing as the engineering team what is the point? If our estimates are constantly wrong and the levels above us ignore them why do we need to waste our time ‘voting’ and playing ‘planning poker’? 

Then there is the issue that in agile we are very loose with stories. Subtasks are added and removed with little oversight. After a story is estimated things can change and suddenly the story goes from 3 subtasks to 10.  The next day 8 of those subtasks are moved into another story. What were we estimating again? 

I will be writing a bunch more on estimation, but some of the things I have gained so far are that expert judgement is considered the weakest means of getting an estimate. The best way is to count something whether stories, square feet of drywall or some other work component. The second means is to use heuristics to compute your way into an estimate. The last resort is expert judgement or gut feel. 

Count > Compute > Judgement

Common Software Estimation Pitfalls

by

Relying solely on expert judgement 

It has been a common experience in my career to sit in a room with a team of engineers and project managers to estimate stories. The typical process is to read through a story discuss it and then ask each engineer how long they think it will take. The engineers guess some number of days, hours or story points. Then the number goes into JIRA somewhere. 

The engineers are experts here, they have done this a lot of times and are much better at it than the average person on the street. But if all you are doing is asking for one number from an expert who isn’t trained in estimation, you will not get good estimates. The most effective estimates come from Counting and Computing with Expert Judgement as a last resort. 

Using estimates in a group setting

Most backlog grooming and other estimation meetings are done in a group setting. Engineers make their estimates one at a time, then the team averages them together. This setup has two issues, first each engineer knows what the previous one estimated and will subconsciously adjust his estimate to match. Less experienced or less confident engineers will completely abandon their judgement and defer to more senior and louder spoken team members. Planning poker is a popular solution to this problem, but it still falls victim to the next problem below. 

The second problem is time. You cannot make a good estimate in 5 minutes. Most estimation meetings run over time because the stories are incomplete and the team ends up discussing them. Each discussion has a risk of going off on a 10 minute tangent. Then with 10 minutes left on the clock the project manager wants to finish estimating 8 stories. 

Not having historical data

I have worked with over 10 different teams in my career. None of them have had consistent historical data going back more than 3 months. A lot of agile estimation assumes teams have historical data they can use to calibrate their estimates. The reality is that teams do not have that data. Either the team does not track both the time to completion and the estimate for a story. Or commonly their is a new project manager every 6 months who implements a new agile process based on story points instead of time estimates. 

To get good historical data you need to have consistent systems for estimation and data collection across teams. Teams are not a good level of granularity for historical data because they change all the time. Teams are dynamic managers change, engineers move to new jobs, ownership of a service moves to a new team. You won’t get consistent historical data if the process changes completely every time your project manager job hops. 

A better way to do Agile Estimates

by

The standard agile estimation process falls into typical estimation traps that make creating accurate estimates difficult. Major traps are  relying solely on expert judgement,  rushing the team to estimate in a high pressure setting, like a sprint planning meeting, and we often estimate work without clearly defining the scope.

When estimating, we want to prioritize Counting, Computing and Judgement, in that order. The best way to estimate something is to have a record of how long things took historically, then create a complete set of stories for the work. To create the estimate you simply count up the stories. In construction you might have a certain number of feet of drywall you need put up in your house. To estimate the cost you would multiply the number of feet by the going rate per foot of drywall and end up with a pretty good estimate. 

In software we work with myriad heterogeneous tasks that we don’t have industry standard data for. So we have to rely on Computing and Judgement to create our estimates. To compute an estimate you could do something like reasoning that since your historical rate of completing stories is 2 days for the company, we will create an estimate based on (2 days * #stories). That is computation will create a rough estimate at best so it would be worthwhile to round it to one significant digit. What you should not do is use your judgement to tweak the estimate after you compute it. Don’t say to yourself, “Our team is the best in the company we will only take 1.5 days per story” if you don’t actually have data to back up that rate of 1.5 days per story.

Our last resort in estimation is to use the judgement of experts to tell us what they think is a reasonable estimate. The problem is that in Agile we rely on Software Engineers who have no training in estimation and our estimating process encourages them to take shortcuts when creating estimates. 

The standard agile process today is to put a bunch of engineers in a room and have them play ‘planning poker’ where they are expected to use expert judgement on a short timeframe, without access to any historical data. 

We want to avoid pressuring people to ‘rush’ when estimating. If data is available we should have time to look it up and compare with previous projects. 

I propose a tweak to the standard sprint planning meeting called ‘Serious Agile Estimating’. Before sprint planning most teams will have a backlog grooming session. After that meeting prepare a spreadsheet of every story that your team needs estimates for. Next fill out the estimation form with the stories for the next sprint and send it out to each team member before the sprint planning meeting. 

The sprint planning meeting will be split into 2 phases, estimation and discussion. The estimation phase will take up the first 30 minutes of sprint planning and should be carried out in silence. Each team member should have a laptop and the estimation form. Team members should take this time to read through the stories creating best, most likely and worst case estimates. After 30 minutes, ask everyone to submit their forms online. 

The discussion phase of the meeting is focused on discussing the differences between people’s estimates. Was there work that we forgot to break up into stories? Do members of the team have differing opinions on how much work different tasks are? What data did team members use to support their estimates? It is important to remember that we do not change estimates during the discussion phase. 

Never make estimates off the cuff or during a meeting while people are talking. Having a hard rule of ‘never make estimates during a meeting’ would be the best. But in my experience it is more reliable to block off focused time in a meeting than to have everyone do it beforehand. 

Serious Estimating Steps

  • Create a list of groomed stories to estimate ahead of time
  • Give each team member a checklist and estimation form (See the example form included at the bottom of this post.)
  • Have each team member estimate stories individually then meet to compare your estimates. 
  • Do not make any estimates during the discussion
  • Require team members to submit a complete set of estimates before the meeting. 
  • Don’t just average the estimates and call it good enough. You need to discuss the differences among individual results, don’t just take the calculated average automatically. 

The big difference is that team members create estimates on their own with ample time to think. No count downs, no poker, estimates are written down and then discussed. 

Finally, I strongly recommend reading Software Estimation Demystifying the Black Art by Steve McConnell which is the source of many of the ideas in this post. 

The book can be found here: https://www.construx.com/books/software-estimation-demystifying-the-black-art/

Google Docs Link:

https://docs.google.com/document/d/1H0CkBlGcRCXxPFMQ3QlXuPzjEUJJRNw/edit?usp=sharing9y9dIw6HnRGm

Story Estimation FormDownload