The trouble with agile is it's a bit too good

So you've gone Agile! Woo! Well done! You've escaped the last millennium's software practices at last! And boy do you feel in control at last! The iterations are flying past, the story points are getting done at a rate you could only have dreamt of. No longer do you wonder what your development team are up to for months at a time, with that nagging feeling that you are pouring money in and you're not getting best "value".

Seems like some kind of productivity utopia doesn't it?

But there's still something not quite right isn't there? Are the technical team still moaning (pah, that's just technical people isn't it? ... or is it? they often have a point, just usually a difficult one). There's this thing they are always going on about, maybe it changes day to day, maybe it's the same. Maybe it doesn't seem well enough defined to deserve a story. Maybe it just some long term gripe that's never quite as important as all those other items in the backlog that have a priority of "O.M.G. if we don't do this by the 14th of this month we're all DEAD!!$$£##£!!!", so it keeps getting barely scheduled and certainly never done, done, done! (Okay, calm down excitable agile people, saying it once is fine.) But hey that's the process so it must be right, if it doesn't make it, it can't have been that important. Can it?

Well, it's time for some unscientific theorizing.

I've been doing a bit of job hopping recently (nothing on the scale of contracting, but I've seen a few different things, and a few different approaches to project management). And I've noticed something in the two examples of agile (SCRUM specifically) I've been close to that bothers me.

“When it comes to the detail of the work, the manager is relying on the expertise of their staff.”

As a bit of background, remember that software development is a highly skilled job. Software development is one of those funny worlds of work where the employee inevitably knows more than the manager. The manager will likely have more broad context (I really hope so, for that matter), but when it comes to the detail of the work, and the right thing to do, the manager is relying on the expertise of their staff to make detailed decisions. This is as it should be given that the developers spend all their working hours, often more, immersed in the detail, keeping up with the current technology, and becoming ever more skilled at the job. Even if a manager is initially just as knowledgeable in the field as their staff, just by virtue of spending more time managing than doing (nothing wrong with that of course), they will inevitably become less knowledgeable than their developers over time (whether they admit it or not, and don't we all know someone who still thinks they know everything!)

Okay, I'll get to the point already (this had better be good).

If you have moved from one of the less well controlled project management methods (including the "general panic" approach), then you may or may not have realised that a fundamental shift in power has occurred. The ability to direct the way your development team spends their time has moved more into the hands of the manager, and away from the hands of the individual developers. On the whole I consider this a good thing, as individual developers deciding to do things by fiat doesn't always help a company with its immediate deadlines, and there is a much improved ability under the new regime to pick a goal and get there more or less on time (unless you are just paying lip service to SCRUM), with fewer nasty surprises along the way (such as, the good old "where did that month go?" experience). In the past, management could give the developers direction one month to the next, but day to day was a bit of a mystery, and without SCRUM in place too it was much overhead to cope with. Now, every day is accounted for. Life is good, the company gets more of what it asks its developers for.

Every developer I have worked with has wanted to do a good job for the company they are working for. And they have all been generally competent at both coding, and interacting with management in getting the job done. So if they are good people, and we are so much more "productive" now, then why were they wasting all this time before? Well, to an extent you can explain the improvements by the elimination of some of the tail chasing exercises that happen under less disciplined approaches to project management. But that's not quite enough. There's something else, and it's a question of perceived priorities and the effect they have on what gets done.

When you have your head under the bonnet, you'll notice all the leaks and all the frayed cables. And as anyone who's taken their car to a garage will know, the mechanic can always find something to suck their teeth about and charge you an extra £150. But when you are just driving the car it all seems just peachy till service time. But why do you give in and pay up for that thing you've never heard of? Surely if the car was fine when you were driving, then it must be okay? Well, I don't know about you, but for me it's the fear of ending up as a pile of tomato ketchup on the inside of my windscreen when I finally find out why that thing I can't even name was actually important. So how is this a good analogy? No-one ever died from bad software, right? Well the point is, the mechanic is skilled (like the developer), and I am not (like the manager). Like the manager, I have to decide which things to spend my money on and which things to pretend I know about and leave till the MOT fails.

Software developers, being skilled tradespeople, always have an eye on the long term, and will always be balancing the current panic from the sales department against what is good for the company in the long run. In the past, when you used to lose months at a time, it was often partly because the developers were taking some time to look after the company's long term interests. In hindsight it is easy to justify the long term work that was done with some glib comment, as it's no longer really up for discussion; you can't get the time back after all. But imagine if all these long term things had to be justified before they were done, even if the manager doesn't know what on earth the developers are talking about. Well you know what, a lot of it wouldn't get done, and the developers who really care about your long term future (i.e. those you haven't ground down yet into despondency yet by ignoring them for years on end), would get narked. If it's hard to "put a business case for" then a lot of good developers I know will just not bother, after all they don't have to save management from themselves, that wasn't in the job specification, and people don't like being saved from themselves anyway. Unfortunately, this is exactly the change that moving to SCRUM introduces. Developers can no longer "just do" something that takes more than a day, no matter how much it needs doing in the long run, as it will be blindingly obvious at the daily stand-ups that they are not sticking to tasks, and are going to make an iteration miss its target. After all each iteration is likely already chock full of "must haves", and even if a developer puts the effort in to get a long term piece of work into an iteration, it will always end up lower priority than user stories for customer visible deadlines, and therefore likely still not get done (unless you are getting your velocity right, which of course you should be).

Interlude. You may now hum to yourself for a bit before I attempt to tell you how to fix it.

So what to do? In a way maybe it's no different to the car analogy. Make sure you (management) get enough long term stuff into the iterations, and give them just as much priority as anything else. Make sure they get done! Just as if they were a short term deadline. Then in the long run, the wheels won't fall off your software, at least not while you're driving. (This blog post contains no warranty, road conditions my vary, any number of factors may cause the wheels to fall off your software. Especially if driven over rough specifications.) Make sure your team of developers know that you are committed to this so that they do actually come forward with the things they know need to be done sooner or later. (If you don't know about it you certainly can't get it fixed.) Perhaps you could create a separate long term plan with the help of the team that provides for long term needs, giving it real deadlines that are as immovable as whatever conference you are showing at next. If you have to justify it to others you can say "because the long term is just as important to our business as the now"! Have the courage of your convictions. Back the long term as well as the short term. Always have an eye on the build up of outstanding long term items (c.f. technical debt). If the long term plan doesn't look like it fits with what you have to deliver day to day, then maybe you need to step back and look at your software architecture as a whole, or the resourcing in your team. I would suggest a practical plan: set a percentage of time that will be spent on the long the term items, say 10%, which is ring fenced for use by the technical experts (the developers), for making sure the long term needs of your software are looked after.

Sooner or later, if things run their new agile course, the chickens will come home to roost, and you'll start to wonder why it's taking longer and longer to get those features out, or more time will be lost to bugs, or things will just start to outright fail. So I urge you to think about the long term and not forget that the manager is not the expert in the detail - that's what your developers are there for. So listen to your techies and the advice they have on the balance of priorities, and take that into account when creating and prioritising your backlog. You will have a happier team and happier software as a result.

So in summary:

• Moving to agile is excellent, but prevents your technical experts from quietly fixing things for you to the same extent.
• Don't forget the long term in the excitement of getting features done, done, done!