Agile Woes

I’ve been working with agile methods for the past decade or so and am a huge fan of “agile done right” (remember that phrase!). In the past couple of months I’ve read a lot of articles where the authors claimed, that:

• Agile does not work
• Is cargo cult
• Is there for managment to control developers
• Is bad for any number of other reasons
• Scrum has too many meetings

I’d like to take a shot at putting the above into perspective. Now, since I’ve mostly been using Scrum this article will focus on that, but I do guess, some of the things will apply to other agile methods as well.

What agile wants to solve

This should be mostly clear to everybody who has had some contacts with agile development, however I’ve often seen a lack of understanding for this in critcs’ arguments. Agile methods don’t want to make software delivery faster. Agile wants to enable us to make the right decisions at the right time. This usually means “as late as possible” (without compromising the project!). Why is that? The writers of the agile manifesto knew, that requirements are bound to change as soon as our customers/users start to interact with the product. This is why agile focuses on early and frequent delivery. This obviously will mean, that the amount of functionality available will be very limited for larger parts of the project. However: This allows us to gain immediate feedback from users, which is a good thing. It might mean, that the users tell us, that our newly implemented feature does not solve their problem and needs rework. With more “traditional” processes this feedback would come at a point where the feature might already be implemented for a couple of months and other features are implemented on top, so changes at that point will be expensive. The reason here is: Using traditional methods features are often designed “ahead of time” and a lot of design decisions are made very early, when the development team (or the architect for that matter) does not know much about the project yet. This project knowledge is crucial and we need to build it for each project anew. Making an upfront design denies us the enormous usefulness of the project knowledge. Agile’s solution here is: Deliver your feature as soon as possible and be prepared to iterate on it. The above is intended to set the stage for the next paragraphs – we’ll have a look at common criticisms of agile methods with respect to Scrum.

Scrum does not work

This is a very common criticism (well not really a criticism, since a lot of people don’t even bother to elaborate further). We know, that Scrum has a number of weaknesses and these really need to be adressed if a Scrum process is supposed to work. I’ll focus on the – in my opinion – major issues:

Short sightedness

The iteration cycle and the perception that is actively promoted by Scrum fosters a certain short sightedness which can get people to produce sub-par solutions in order to get “the sprint complete”. Coupled with the fact that Scrum does not include architectural work the “emergent architecture” phenomenon will cause maintenance problems in the long run.

How to avoid:

• Nobody says you’re not allowed to have work items that deal with architectural and/or conceptual topics. Having these WI prior to the implementation helps to create a shared vision for a feature and also will make future rework less likely if the conceptual work is used to obtain very early feedback from stakeholders. We also get the possibility to see how the new feature fits into the architecture as a whole.
• Architectural sketches: In most of my projects we worked with architectural sketches, to lay down the basic architectural building blocks of the project early, so that other features have something to integrate with. This might sound somewhat contradictory to the “as late as possible” idea, however, if done right the architectural sketch should be flexible enough to evolve and at the same time give enough structure that allows to avoid the most grave mistakes.
• “Vision Keeper”: While Scrum does not have the concept of a lead developer and I don’t want to inject it, it is very helpful to have one or multiple persons on the team that really understand the architectural vision and can judge new concepts based on that. Very often these will be senior developers who will already have a lot of say in most technical discussions.
• At last: The sprint is not sacred. If the team does not meet the sprint goals this is somewhat annoying, but the world will not end. If a story needs more work to be ready then don’t be dogmatic and take it to the next sprint.

Estimations and heterogenous teams

Scrum assumes homogenous teams where each team member is able to mostly perform the same tasks. This is sadly often not feasible. As an example one of my more recent projects consists of a largeish complex C++ codebase for an embedded device and on top of that a medium sized angular web app. The team working on this project is divided into webdevelopers and embedded developers. However they share the same backlog. The bad: Depending on the backlog contens velocity will vary between sprints a lot stronger than is usual with homogenous teams. This leads to poorer predictability. The usual calibration (which is said to take 4 – 6 weeks in scrum) takes decidedly longer (for the team in the example it took around 10 weeks to reach a state approaching a stable velocity).

Scrum is cargo cult

Sadly, very often this is true with bad implementations, where the team does not understand scrum well. In that case we often see, that they follow procedures but don’t gain the benefits because they don’t understand how to reap them. What helps in these instances is probably thorough coaching by an experienced agile/scrum practitioner.

Scrum is for management to control developers

This one is mostly raised to discredit dailies, and I can somewhat relate to that, as I will occasionally drop in on the dailies of my teams as well. I guess there could be an impression that management wants to see what developers are doing, and that they’re not twiddling thumbs. The perception of the critics is usually: “Management wants me to justify what I’m doing.”. I feel sorry for that, since the critics that voice this oppion will probably have had a matching experience. So, why do I “control” my teams by dropping in on their dailies? Well, I don’t control them. I don’t want justifiation. I want to get a feeling for the projects, so that – if something goes wrong – and I am asked to intervene in some manner I at least have an idea what is going on in the project. Being in a daily is – from the perspective of someone not working on the project – barely scratching the surface. One will notice if there are troubles or if things are chugging along smoothly. To get this straight: The purpose of the daily is to distribute information among the team members, that allow the team to:

• Clear out road blocks, e.g. if developer A is stuck with a problem and developer B has worked on the feature or area of code before the daily is the chance for these two developers to band together to solve the current problem.
• Avoid conflicting work, e.g.: If developer A is working on a critical piece of code, mentioning this fact should (at least in theory!) cause the team members to sync with that developer, should they need to make changes to the same code.
• Tell the rest of the team, when someone else needs to take over tickets. While everyone should look at the board for new work, more often than not tickets will remain in limbo after the main development was done.

Scrum has too many meetings

Scrum prescribes 5 types of meetings:

• Dailies
• Plannings
• Groomings
• Demos
• Retros

Of these only dailies are, well, daily. The others are often only once per Sprint and the grooming meeting is optional. Assuming a two week sprint this leaves us with 13 meetings for the process. However: I’d argue that the number of meetings is less of a problem than the time spent with the meetings. So, if we assume that dailies are short (2 min per team member max!) the dailies should not take longer than 1.5 hours over the course of the sprint (as an example: One of the teams with wich I actually still do project work has five members and we regularly need less than 10 minutes for our dailies). Plannings are another story. If the backlog priorization was done right and the workitems are in good shape a two week sprint can be estimated and planned in about an hour. However: If the backlog needs work and that work is done during the planning, these meetings can easily escalate into hours long marathons which add little value. I’d usually argue for not working on individual workitems during the planning (except minor changes, if clarifications are needed, but these should be the exception rather then the rule). If stories show up that need work before they can be estimated it’s probably better to push those to the next sprint, instead of wasting everyone’s time. At that point we should also be aware of the costs meetings cause for the organization. And long plannings are especially bad in that regard: Each minute spent costs NumberOfParticipants man minutes, while usually only one or two people will work to flesh out workitems, if that is done during planning. This leaves us with demos and retros. If we allocate one hour each we end up with about 4.5 h of scrum meetings per sprint (or just over 5% of the sprint’s working hours), which actually seems fairly reasonable as far as process overhead goes.

The points mentioned in this chapter are – as stated – the major problems I see with scrum. There is more to this and all things considered I’d say Scrum is very hard to “get right”. In fact I’ve seen more bad implementations of Scrum that did not work than good implementations that actually worked as advertised. And don’t get me started on trainwrecks like “SAFe” (the A is a blatand lie!). The few good implementations of Scrum I’ve seen in the last decade worked really well and the teams were able to deliver very predictable performance most of the time. However, as stated, most implementations were pretty bad, with their “badness” mostly stemming from doing “scrumthing” without really understanding it. The pathological failures were – among other things:

• “Management does estimations of stories”
• “We determine during planning who will do which story”
• “Our stories consist of headlines only and developers and testers keep arguinng about what actually needs to be done”
• “We don’t do estimations”
• “We don’t do dailies”
• “We do dailies with 40 people and keep wondering why they take north of an hour.”
• “Sprints are filled eight weeks in advance with no wiggle room whatsoever”
• “We don’t do commitments, our techlead tells us what we do in the next sprint”
• “We don’t measure velocity”
• “We write our stories during planning, our planning takes six hours”
• “We don’t have acceptance criteria”/”Our stories are mostly open ended, so nobody really knows, when the story is ‘done’.”
• “We don’t need retrospectives”

Typically a bad implementation ticks at least two to three boxes above, and this is usually enough to break scrum in a very bad way causing the organization to suffer all the bad parts of scrum while having none of the advantages. Typically the problems appeared due to outside intervention (i.e. “Management”) not letting the team work. One of the things one has to understand is, that each of the “things” contained in scrum, be it retrospectives, demos, plannings, dailies, backlogs and whatnot serve a particular purpose. If we change those things we better know what we’re doing, because we’ll -in some way or another- will have to accomplish what the scrum “thing” accomplished.

TL;DR

Most criticisms of agile I’ve come across take aim at bad implementations or arise from a missunderstanding of the ideas behind the repsective methodology. As stated, there are lots of bad implementations in the wild, which is not surprising, given that implementing – e.g.- scrum correctly is very hard, especially since a lot of the time the development teams don’t control some aspects of the organization’s behavior, that are critical to agile’s success. I’d argue that – in these cases – the organization as a whole might be better off using a non-agile approach, rather than the faux-agile they end up with if they try to – unsuccessfully – adopt agile practices.