One of the things that you’re not going to find in this seven blog posts series on measuring the software development process is measuring velocity. I hate velocity because I’ve seen it misdirect managers and team members far more often than I’ve seen it provide valuable information.
Rather than spend time teaching teams about story points and velocity, I much rather show teams how to break features down into small homogeneous tasks that can be achieved in less than four hours. I know that some work doesn’t lend itself to be done in small chunks but in most situations I’ve been able to find ways of breaking down tasks into small chunks and I find that it’s very valuable when I do this.
If every task is a small, manageable piece of work that can be done in less than four hours then we don’t have to assign story points to tasks because everything is a small. Now we ask how many tasks can we do in this iteration and very likely that number will be similar to the number of tasks that was completed in the last iteration or perhaps even the one before that. This is usually more than enough fidelity to get a sense of how much work the team can accomplish in the present iteration.
Story points and velocity were designed in a way to be fine tuned for accuracy in short time increments and as a result, velocity cannot be compared across different teams and it cannot even be used to compare the performance of the same team over different time periods because the very meaning of a story point shifts for the team through time. The team may still estimate that they can finish a hundred story points in an iteration but the amount of work that that hundred story points represents today will be very different then the amount of work that it represented six months ago for the same team.
This actually improves accuracy in the short term but it means that velocity can’t be used for measuring productivity through time or across teams and so we like to say that velocity is not a productivity measurement, it is a capacity measurement. Rather than dealing with all the caveats and addendum’s related to velocity let’s just throw it out and stop tracking it.
But there is one statistic that I do track across different teams and through time because I find that it is scalable across teams and through time. The metric is defect density.
I define a defect or error or bug an incident or problem that escaped into production. Some teams track internal errors and if there is a separate quality assurance phase then this could be helpful for measuring the effectiveness of the QA effort but often I will opt just to look at defects that escape to production. I will rarely look at the severity of the defect and treat all defects as equal.
When comparing defects across product lines or across teams, it may make sense to look at the impact of defects or just the number of defects across product lines but often I find the fairest way to compare defects across teams or product lines is to look at defect density which I define as the percentage of defects per thousand lines of code. This helps normalize comparisons against small projects versus very large projects.
Of course, defect density is a powerful measure of the effectiveness of a software development process. When bugs consistently escape to production it tells us the something is seriously wrong in our software development process. Teams who adopt test-driven development and the other Extreme Programming practices often see a huge drop in defect density. Defects are expensive so dropping in defect density also drops costs. Reducing defect density can be a big and early win for teams when adopting Extreme Programming practices.
Note: This blog post is based on a section in my book, Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software called Seven Strategies for Measuring Software Development.
Previous Post: « Measure Time Spent Coding
Next Post: Measure Time to Detect Defects »