Technical debt isn't actually formally recognised by Agile or Scrum, and in my experience the ability of development teams to actually recognise technical debt is very patchy. There are people, teams and entire organisations that experience technical debt without ever seeming to understand it, or perhaps without wanting to acknowledge it.
So the challenge of technical debt is not just to provide specific ways to manage it, it's also to understand how to recognise it in the first place. Let me take a stab at a working definition :
Technical debt occurs when you create a solution using techniques which satisfy the immediate need, but which create problems for the implementation of future requirements.
Note very carefully that technical debt is not the same as a bug in your code. Let me spell out the sequence of events that I'm thinking of for the purposes of the above definition.
1. You devise a solution to a particular requirement which meets your definition of done - it passes design review, it meets the coding standard, it passes the tests, it works in practice.
2. A few months later, a new requirement comes along.
3. You cannot implement your preferred solution without changing the ( non buggy ) code that you created in step 1. This would have widespread knock on effects, and the amount of effort involved would be quite large.
4. You decide to avoid changing the step 1 code, and instead satisfy the new requirement by means of a non-optimal solution that works around the problems posed by the step 1 code. Now, and only now, you have technical debt.
Because the business value of technical debt is negative, but this negative value is only realised when some particular positive feature is being developed later on, it is hard to grapple with.
For developers, there's a very obvious conflict between Agile's emphasis on working in the 'now' rather than attempting to foresee future requirements in detail, and the risk of creating technical debt by not considering possible futures.
One particularly dangerous response to technical debt is to think 'if I create very general and flexible code, technical debt will be less likely'. Unfortunately this is true, which is why it's tempting. But there are two problems here.
Firstly, you do not have a crystal ball, and no matter how hard you try there will be new requirements that can catch you out. But secondly, and more importantly, maximising the generality and flexibility of something requires very careful thinking ( i.e. a lot more effort than just addressing the problem in front of you ), and also maximises the testing effort. If you know this flexibility will be needed, OK. But you don't !.
It could be argued that this is an architectural concern, and in some environments that would mean this 'consideration of the future' was taken care of by architectural work done prior to any implementation. But this is not a complete solution - architecture is as prone to creating technical debt as implementation, and architecture will not usually specify implementation details.
Possibly it is a Product Owner concern. Often the reason for technical debt is that time pressure or budget constraints affect the choice of solution, and these are things which suitable product backlog management and iteration planning could mitigate.
However, this places a premium on communication between the team and the Product Owner, as it is not a foregone conclusion that the Product Owner would have the technical understanding to anticipate and resolve this type of problem.
Let me step back for a moment. In most circumstances, it would be hard ( though not impossible ! ) to create a significant technical debt in the space of a single iteration. So technical debt is something that tends to build up gradually.
A consequence of this is that it can be managed gradually. A mistake which I feel is often made is to put off dealing with technical debt until the burden has become intrusive, at which point it has actually been affecting productivity for a significant period of time. This loss of productivity can be difficult to recognise. When it is recognised, this then translates into a desire, or need, for a 'big bang' technical debt fixing effort, which derails the regular cadence of value delivery that is one of the benefits of Agile.
One project I've known tried to recognise this problem explicitly, and when planning each iteration considered whether to add explicit 'debt recovery' stories. However, the problem tended to be that the Product Owner would have difficulty evaluating the business value of such work, so these stories would get a low priority, until suddenly a higher priority item required a fix to make it work, and hence we're back to a reactive pattern.
But again, the perspective here is unhelpful. By 'perspective', I mean the idea that dealing with technical debt can somehow be separated from 'normal' development.
What if, instead of throwing all your technical debt in a bucket to be dealt with later, you deal with it as you encounter it ? It's not that there would never be technical debt in your product - but any technical debt that was recognised during implementation of the 'now' requirement would be removed as part of that work, instead of being worked around, passed over or otherwise left for later.
In this way, the timescale of technical debt removal matches the timescale of new value creation. Technical debt becomes a 'steady state' aspect of your product - never entirely absent, but never so great that it presents a major impediment to change.
Of course, the astute observer will notice some possible issues with this approach, the main one being that if the debt is not recognised at sprint planning stage, then estimation will be affected.
Technical debt can take several different forms, not all of which are amenable to being resolved simply by altering the code that you have written.
Vigorous refactoring is often discussed in the context of technical debt, as a means of mitigating it. However, valuable though refactoring can be, it is not a magic bullet.
Perhaps your product has chosen to use a third party solution for some part of it, and your code is written around that choice. A deficiency in that third party code cannot be eliminated by altering yours. There may be cost and contractual matters to take account of if some new supplier must be used, or you will at least have to wait for your supplier to issue a fix. So you are stuck with mitigating this 3rd party deficiency in your code ( i.e. incurring technical debt ) until a commercial solution is found.
So it is certainly the case that not all forms of technical debt can be dealt with 'in line' as and when they are encountered. In the situation described above, it may be possible to plan the introduction of a solution via normal Product Backlog management processes.
My expectation is that most of the time competent developers would not simply stumble across some unknown technical debt as they were working on a coding task. For a start, it's often the case that developers know when they may be introducing technical debt - i.e. it's easy to recognise step 4 in the description above. It may even be useful to keep a record on the Product Backlog whenever this occurs. Secondly, when developers engage with the Product Owner for backlog refinement purposes, they should be considering potential technical debt issues.
However, it's reasonable to ask whether dealing with technical debt should reduce the velocity of the team. This is tantamount to asking whether removing technical debt adds value to the product. The key to whether technical debt removal adds value is the word 'debt' - thus removal of the debt is a pure cost, and does not add value in itself.
On the other hand, technical debt is often incurred as a result of a deliberate, conscious trade off between implementation effort and the desire to deliver the simplest working code that has business value. 'I can see a better way', the thought goes, 'but it will take half a day - it's not worth it, I can meet the DoD/Acceptance criteria with what I have'. Should a team effectively be penalized for successful delivery of business value, when it simply is not known whether or not technical debt will be incurred later on ?
In practice, what I observe is that designing and coding in a manner that minimises technical debt is a learned skill. It is not inevitable that any code you design or write must incur technical debt of some sort. However, as normal, without feedback to tell you when you could have done better improvement is difficult.
On balance, I favour treating technical debt as if it were defect fixing - i.e. the more technical debt a team incurs, the greater the hit on their velocity. The principle in play here is for this to act as encouragement for the team to learn techniques for minimising technical debt, having due regard to root causes.
In effect, I therefore expect technical debt to inflate the size of stories without increasing their value.
Yes, technical debt is hard 🙂. But it is not hopeless. Agile practice is really quite well suited to dealing with technical debt, and in particular avoiding the problems which occur if it is allowed to accumulate.
In the case of Scrum specifically, the Transparency and Inspection principles mean that the existence of technical debt is noticed, and the need to manage it is recognised. The Adaptation principle empowers the development team to make the required changes.
In general, a hybrid approach to managing technical debt will be needed, because of the multiple forms that technical debt may take. Where the technical debt exists in the design or implementation, an in-line approach will often work well. Where the debt exists due to higher level problems, explicit management via the Product backlog may be necessary. Technical debt should impact velocity, in order that it becomes visible and can therefore trigger adaptation.
© Mark de Roussier 2022, all rights reserved.