In this installment of our re-read of Actionable Agile Metrics Volume II, Advanced Topics in Predictability we again tackle two chapters. Chapter 11, “Different Sampling Methods”, points out that random sampling techniques are just the tip of the iceberg in approaches. Chapter 12, ”What Percentile Do I Choose?”, discusses whether you always need to quote the 85th percentile range. The short answer is no but like most things there is more to it than that.
Chapter 11 – Different Sampling Methods
Monte Carlo Analysis is a powerful technique for probabilistic forecasting – Can we agree that is a given? What data is fed into the Monte Carlo formulas uses a sampling technique. For example, assume a Monte Carlo Analysis of 1,000 executions on throughput for a team’s last 10 two-week sprints including all 140 calendar days in the data set (10 * 14 calendar days per spring). A random sampling technique would randomly select from the data set 1,000 times. A Sunday would be as likely to be selected as a Tuesday. Most practitioners use the random sampling approach, if for no other reason that it is easier to code. Tools like Actionable Agile from 55 Degrees (I highly recommend this tool) use this approach. However, randomization is not the only approach.
In Chapter 11, the Author and Prateek Singh compare six approaches. For fun, I asked Gemini (Google’s AI) for a list of Monte Carlo sampling techniques and it suggested several others. The bottom line is that the “best” Monte Carlo approach depends on the problem you’re trying to solve. Consider factors like the target distribution, desired accuracy, computational efficiency, and available information when choosing a method. In most scenarios, the simple random approach will suffice (the authors suggest that the outcome is better in most cases); however, I am intrigued by the “importance” sampling technique mentioned by Gemini. Note: until I do more research and experimentation this approach might be similar to the Weekdays and the Weekdays and Weekend approaches tested in Chapter 11.
For those who are not Monte Carlo geeks the real message of Chapter 11 is that probabilistic approaches like Monte Carlo use the past to predict the future. To quote the Author(s) “One big assumption that all Monte Carlo Simulations (MCS) make is that the future we are trying to predict roughly looks like the past that we have data for.” This means that we have to think carefully to pick a period in the past that “we believe will accurately reflect what will happen in the future.” The downside of this statement is that it seems to give those doing the Monte Carlo carte blanche to pick the past that they want. I strongly suggest care be taken to consider context, to think hard about excluding “rare events” and to be VERY transparent about the past you are using to predict future events. Finally in the Author’s words, ”you are much better off investing in more consistent throughput for your process than in trying to come up with more and more sophisticated MCS models.” Enough said!
Chapter 12 – What Percentile Do I Choose?
When you deliver a probabilistic answer to the ancient question, “When will you be done?” or its cousin, “How much will you complete?”, what range and percentile of confidence do you present the answer with? I tend to use the 85th percentile at my guidepost, without much thought. If I was answering in terms of time an answer might sound like, “We get requests done in 15 days or less 85% of the time.” Note: I would only say something like this if the piece of work did not look out of line – context is important. Opening Chapter 12, Vacanti suggests that there is an idea that the 85th percentile must always be used, which is not the case.
What struck me when reading Chapter 12 was the idea that continuous forecasting allows you to tighten your forecast range. Shortening the feedback loop makes the process self-correcting. In organizations, I have worked in and with over the years, re-forecasts and reports tend to be tortuous monthly affairs or, for really large efforts, quarterly dog and pony shows. Wider ranges protect the program manager or product manager from having to explain changes. Vactanit holds up NOAA’s National Hurricane Center’s forecasts (https://www.nhc.noaa.gov/ – this is one of my favorite sites during hurricane season) which are updated constantly (and published multiple times a day) as a gold standard. An example of the forecast cone (70th Percentile) is shown below.
If drawn to include all possibilities up to the 85th percentile the cone would be much wider and provide less value. A lot of people who did not need to worry would be diving for cover.
Continuous forecasting reduces the need to be overly conservative which makes it easier to see change. And as importantly to be able to react to it sooner when you need to.
Logistics Note: I believe we are two to three weeks from the completion of our read of AA. Thoughts on the next book?
Buy a copy and get reading – Actionable Agile Metrics Volume II, Advanced Topics in Predictability.
Week 1: Re-read Logistics and Preface – https://bit.ly/4adgxsC
Week 2: Wilt The Stilt and Definition of Variation – https://bit.ly/4aldwGN
Week 3: Variation and Predictability – https://bit.ly/3tAVWhq
Week 4: Process Behavior Charts Part 1 – https://bit.ly/3Huainr
Week 5: Process Behavior Charts Part 2 – https://bit.ly/424O5Wc
Week 6: How Much Data? – https://bit.ly/47GVP24
Week 7: Detecting Signals – https://bit.ly/3SjwfdO
Week 8: XmR Charts and the Four Basic Metrics of Flow – https://bit.ly/48j5AU9
Week 9: Myths and Other Considerations – https://bit.ly/3SvfgVU
Week 10:VoP, VoC, Predictability, Monte Carlo – https://bit.ly/3UGdxzT
