How’s Your Project Going? – Metrics for Development Builds

, Sunday, January 25th, 2009

Few things scare clients like the “big black box” of development.  At the development phase, after intensive iterative rounds of creative and functional decisions with a tangible sense of back and forth, the client puts their faith in their consultant.  How do the client and consultant determine how well their development project is proceeding inside of that black box? Metrics – and the basic metric, understandably, is about budget.

Every time I discuss metrics, I realize that as a Tech Lead, my goals are not quite the same as a Project Manager – even Technical Project Managers.  Timeline, scope, budget – that’s what my PM is obsessing over.  Developers absorb the same concepts in a different way, through the classic adage “fast, good, cheap – pick 2.” But those demarcations make the most sense during discovery or design estimates.  In the middle of a build, things change.  My job is to make sure the build gets done no matter what, and my PM has to worry about it being paid for.  Scope is still vitally important, but budget, as it relates to profitability, is nearly meaningless to a Tech Lead, much to my PM’s chagrin.  It’s not that I don’t worry about the client’s budget, but that during development, the financial consequences of rates and margins are distilled down into staffing – how many developers of what skill for how many hours per week.  And timeline becomes the inevitable algebraic solution of when that staffing meets scope (with a little bit of magic pixie dust thrown in to the equation).

Recently, I was discussing metrics with a PM after a major build and had trouble articulating my thoughts.  It seemed to me we only achieved real clarity about tracking at the time when we had the least time to implement it.  A few days later, I came across a detailed email I’d written during the heady height of a previous build and completely forgotten about.   Not only did it address improvements to our current metrics, but it linked them all together to judge project health.  It was like reading the penned version of dream before some guy from Porlock wakes you up and everything turns vague and indistinct.  It was fantastic.

I couldn’t understand a word of it…

A little bit of clarity and some pretty charts, after the jump.

I couldn’t understand a word of it … which probably explains why no one replied to it at the time and it spent 6 months languishing in my Sent email box before I remembered it again.  Part of the problem with metrics is the sheer number – qualitative metrics, quantitative metrics, metrics based on profitability, accuracy, etc, etc – but also trying to apply them in a meaningful way to achieve your goals.  In dissecting my email, I realized my problem was that I’d explained all the metrics, what they represented and how they’d let us adjust things to make development go smoothly.  What I’d failed to do is to tie the metrics to the questions we asked ourselves during the dev process.  With that in mind, I’ve created a chart that details a simple 3 metric development process for Tech Leads to follow, starting with the questions and ending with outcomes.

What’s going on here? Well, first of all, the base playing field is split between scope and staffing, both of which overlap in the middle.  Those are your malleable allocations.  Where’s timeline?  Let time take care of itself. If your burn rate metrics say you won’t make your launch date, it’s no good just moving a target on a project chart.  You need to adjust your staffing to get more done in a faster amount of time, and you can calculate your timeline from that.  Or you need to adjust your scope, which means you’ll have to adjust your staffing, and again – you can calculate your timeline for that.  The process is the same for timeline extensions.

Next, notice that the process is built into a repeatable flow: asking questions, analyzing metrics, and then altering outcomes.  There’s a branch dealing with scope, one dealing with staffing, and another dealing with holistic project progress that falls between the two concepts.  Each branch has a single metric that allows Tech Leads to make some smart interpretations of their project progress.

The chart details both the questions and outcomes each metric is involved with – here’s info on the metrics themselves:

Scope – The Task Progress Metric

At its most basic, task progress judges the development team’s estimate of percentage done on a task with the actual hours billed to it.  Simple scenario: Task A was estimated to take 100 hours.  Your billing indicates developers have already done 80 hours, but they estimate they’re only 50% done with the work to complete the task.  If you project that out, the final development hours is likely to be more like 160, or 60 hours over the estimate.  Ideally you’d be tracking each task in a development build, which would allow you to project out the final development hours for the entire build, and come up with a calculated completion done % for the entire project.

Most development teams already do task progress percentages, but the most comprehensive Task metrics incorporate:

  1. Frequent analysis during development – depending on the project type, anywhere from every week to every day
  2. Correlations with actuals – a must for actually validating the developer’s guesses
  3. Detailed tasks lists – task descriptions narrow enough to provide feedback on
  4. Integration with timesheet systems – to enable metrics to be run promptly and accurately
  5. Weighted overage/underages – to demonstrate the relative importance of specific task inaccuracies versus the entire scope of the development
  6. Established precedents for completion levels – checkpoints that prevent tasks from getting above a certain percentage when QA has not been performed or the client hasn’t signed off

Staffing – The Burn Rate Metric

Burn rate is an analysis of how quickly your developers are “burning up” the hours assigned to them spread across the timeline of the project, done individually or as an aggregate average of all the developers on the project.  The concept is essentially similar to its dot-com era namesake that reflected how quickly startups were burning through their capital, only here the capital is the total development hours available for the project.  If you’ve targeted your developers to do 50 man hours of development each week, and 3 weeks in they’ve only managed to dedicate 75 hours over the entire stretch, they are only burning at 50% of the expected rate.  A proper burn rate chart should use data from both booked and actual hours and calculate a new timeline for you based on the burn rate, or provide new burn hour targets to accomplish the original timeline.

It’s dangerous to try and make a burn rate chart do too much – burn rates are inherently simple mechanisms that idealize development,  pretending that developers do roughly the same work at each stage of the dev process and are burning hours as if the task estimates were 100% right.  Most importantly, in their simple form, they do not represent the health of a project’s budget – a team of programmers might have done 50% less work than expected at a certain point, but they might also be 75% done in terms of tasks.  A more accurate burn rate metric might adjust staffing to agree with completion rates to give weighted numbers.  But go back to the original questions that a burn rate is supposed to answer – does staffing need to be adjusted, does the timeline seem accurate, etc.  Both the task progress and project progress metrics are better suited to answer questions about the project’s completion state, so use the burn rate for its strengths.

Holistic – the Project Progress Metric

The project progress metric is possibly the most basic analysis you can do on a project, as well as the one open to the most interpretation – and every project team uses this metric, whether they formalize it or not or use a completely different term to describe it.  It’s nothing more than an estimation of the entire project’s completion percentage, and the source of the estimation can come from a number of areas, including milestones hit, experience with similar builds, and sometimes — pure guesswork.  It’s important that it not be a calculation of task progress or burn rates in staffing, as project progress metric is a separate concept that can serve to reinforce or invalidate those other numeric analytics.  A single bare number (% complete) is an essential number to be used to project out additional hours (or fewer – lucky you!) needed to complete a project, providing additional data you can use to run through staffing or to force scope change.  It’s a number based on experience, and liable to be debated and cause disagreements.  That’s a good thing – there should be plenty of room for discussion because this metric is so crucial to the process, and because it is an aggregate of so many distinct parts of the build.

Three relatively simple metrics can put the whole build into perspective, put development back on track, and keep Project Managers and clients happy with numbers instead of complaining about the “black box” of development.  A few final words of advice on utilizing build progress metrics:

Start With Questions – they define what metrics you need and what you should get out of them

Use the Metrics You Need for your Role - don’t expect that the PM or the Operations dept has the same indicators you need to do your job

Alter Your Plans – don’t just use the metrics during post-mortem to tell you where you went wrong – make adjustments midstream to ensure a successful project.

No Substitute – metrics are never a substitute for experience and subjective interpretation during a project, and the best ones server to validate the Leads’ hunches with analytic data

Comments are closed.