Metric Driven Design
Metrics; measurements with purpose. It’s a term I’d long left associated with those business managers that trade buzzwords with each other; inheriting, instead, the engineer’s standard unit- a deliverable. And deliverables are important; key performance indicators of beams designed per-hour or daily count of piles driven (OK, I’m being a bit facetious) are nothing compared to handing over a complete and functioning bit of infrastructure.
Last week I went to a talk from an enthusiastic young man from everyone’s favourite online takeaway service. The topic- Metric Driven Development. It’s fair to say that software developers have a real fascination with building and analyzing their processes; and in my short working life I’ve already seen the rise (and somewhat fall) of Test and Behaviour (Driven Development) approaches. But that’s what makes it so interesting…
So what is this metric driven development? Would it surprise you to learn that you’re already part of it? You click on the order button- was it always on the right? that took a moment or two to find- and stall for a bit at the checkout; unsure on whether you really need everything in your basket. But there’s that timer going; seems this deal is only for a short while. Best click through. Ahh, this site never remembers you payment details. The saving isn’t that great in any case- let it go. Have you seen that video with the hedgehog in the bath?
Every single step of that process has been measured (assuming this online store is using metrics). And someone is defining insights; goals. They know that 20% of people, like you, have dropped out at the last hurdle. And so they launch a few campaigns to inform a decision- 10% of visitors are now being offered PayPal as a checkout, and another 10% are being offered to save their card details. The monitoring continues- it turns out just offering to save the details only increases conversion by 5% (perhaps people resent all data entry?), but that PayPal link improves things by 10%. Slowly but surely all visitors are being invited to use PayPal…
There’s a lot of talk about in this example. Firstly let’s establish that their ultimate goal is to convert you from a browser into a sale. The metrics they collect are used to inform their design. They’ve targeted aspects that they think can inform decisions; not only watching your journey but also the speed at which you make it. In a previous incarnation people once gave up after spending too long considering their choice at the checkout- thanks to the timer, they no longer do this.
Although people slow down a bit finding the right buttons, the point where they lose the sale is when they ask for payment. So, following a development driven by metrics, it makes sense to target this aspect first. However it’s hard to know exactly how to solve this problem, so they prove their solution by doing something called A/B testing. They try both of the solutions they’ve considered, and see which one performs the best. Building an architecture to store payment data, is expensive so you want to be sure before you invest.
The last manifestation of the metric driven approach is more subtle. Despite PayPal being a clear way to improve conversions- they didn’t just roll it out in one day. Using an approach called feature toggling, they slowly switched it on for users. This allows them to use performance metrics to ensure they don’t bump into any problems; constantly verifying the health and load to inform decisions on how to scale it up from 10 to 100% of their customers.
I hope that the advantages of metric driven development are fairly clear. You get a very clear goal to inform every decision you make; I want to increase the measure of X, will Y do that? It follows a very scientific principle; hypothesise, experiment, evaluate, respond. Failures can be spotted faster, as can changes; by integrating instrumentation it’s easy to spot trends, and verify that what you put in to achieve something continues to do so.
One thing is for certain, civil engineering isn’t software engineering- however I don’t think that means that ideas are mutually exclusive. In my last post I talked about cost and consequences, and hinted at a need for instrumentation (that’s just the technical term for measurement equipment) as part of our infrastructure [Ed. There was even a comment about it.]. These days instrumentation is cheap, as is data processing.
There’s a few spaces where a metric driven approach might help construction. Starting at the most mundane, simply measuring how long it takes to achieve something and setting it as a metric to reduce can be surprisingly powerful. Sure- everyone in construction wants a fast turn around, but unless you make a formal statement of this you’ll just be using anecdotal balances and it’ll be hard to justify and discover exactly what you did that made it so fast/slow.
However, management theory aside, I think there’s some more exciting applications. Consider our infrastructure as one giant system. The goal is to get people from A to B as fast as possible. How do we evolve that system? Every new junction provides an opportunity for massive scale A/B testing– each with it’s own unique feature that either lives or dies based on the relative improvement. Measuring traffic flow is easy; it’s coordinating the results that will be the hard part.
Or one step further. A structure built of metrics. A bridge wired with sensors. Using the responses from the sensors to evolve the design– sending across the first car, measuring the response and strengthening accordingly. Slowly rolling out to more and more traffic and making the adjustments as discovered- entertaining new prospects through the A/B pattern; toggling structural aspects to manage their effect. Sure, a more modular construction approach would be needed, but I wouldn’t be surprised if you ended up with a more efficient structure that was easier to build (as it had to be built as we went along). And, of course, you’d learn a lot.
It’s clear from my most extreme example that the approach is more of a science than a traditional engineering one. Alone it’s much more of a fool doing for a pound than a precision penny; lots of guesswork and learning through feedback. However the role of the engineer in this scenario is to interpret the metrics and suggest appropriate responses. Something, you might argue, that your average civil engineer already does!
Of course, in our current system this is a bit of a pipe dream, however as a thought experiment it’s an interesting one.