Where does design fit into Agile, and in particular Scrum ?
The reasons why this is a question at all are rather complicated.
I'll start by baldly stating where I intend to end up - in any complex, multi-feature product, there is a level of design which is a prerequisite for developers to create working software. In Scrum terms, 'prerequisite' means 'this is not something that is done by developers during the Sprint '. Instead, it is available for developers to refer to as they do the work for the specific product increment they are delivering. It is an input to Product Backlog refinement.
There are many if's, but's and maybe's buried in the above statement, and I'll try to tease these out as I go.
The whole question of architecture, and how/when it should be done, is central to the Agile scaling problem, and is often discussed in great detail when considering scaling issues - see the SAFe take on the issue here for example. Although I've taken pains to base my views on my experiences, I draw a certain amount of comfort from the fact that I seem to be roughly in agreement with significant chunks of SAFe and similar tools.
The concept of design frequently features in descriptions of a waterfall development process. It's the part that comes in between requirements capture and implementation. So we have something like :
Requirements Capture -> Design -> Implement -> Test
In actual fact, Agile is no different. It's simply that instead of :
Capture All the requirements ->
Do All the Design ->
Do All the implementation ->
Do All the Test ->
Yay, Product !
we have something like :
Pick a few of the most valuable requirements right now ->
Create the tests for those requirements ->
Do enough design for those requirements ->
Implement the design ->
Test the implementation ->
Yay, Product ( or part thereof ) ->
Repeat.
In other words, it's all about scope limitation, timescale, and iteration.
But the thing that stands out in all of this is that design is what is implemented - you do not implement a requirement other than via some sort of design function. It may be a complex function, or it may be tossing a coin between 'do it like this' and 'do it like that', but it's difficult to avoid.
So why does this present a challenge ?
Note that in the description of waterfall given above, all the design happens before implementation. Understanding the Agile problem starts with the realisation that there are different types or levels of design. In particular, there is the concept of Architecture. The creation of Architecture is a design task.
At a practical level, what I mean by 'architectural design' is any layer of abstraction above that of entities supported directly in the implementation language. So I don't mean 'the Wurble class is designed using Boost::Regex'. That's 'code design'. I mean 'There is an entity called Wurble that provides an interface for filtering user input according to context specific criteria. It is a Restartable entity exposing the Restart interface, and has a Session manager instance'. Or some such.
Architectural design has a purpose. In a nutshell, an architecture defines an abstract system of components in terms of their behaviours and interfaces. The level(s) of abstraction employed in the architecture may vary, but the purpose of such a definition is to ensure that all the functions required of the system as a whole are provided by the system in as simple and efficient a manner as possible ( but no simpler 🙂 ).
There will very likely be several different perspectives from which the same underlying architecture can be viewed. By implication, architecture is concerned with ensuring that all user perceptible system features can coexist happily. The need for architecture grows as the scale and complexity of the system grows.
The above just barely scratches the surface of an enormous topic ( e.g. see this book if you really must 🙂 ), but it suffices for the argument that follows.
Let's turn the idea round and see what that shows us.
Let's imagine that a system ( a Product, in effect ) is being created with 10 main features. Some interact directly with each other. All are sufficiently complex to require domain knowledge in their own right. The company has assigned two Scrum teams to work on this product. No architectural thinking or design has been done.
Team A starts work on a couple of stories from Feature 1. Applying the 'shoot a bullet through the stack' principle, they decide they need a little bit of UI, a slice of a middleware thing that interacts with some corporate resources and assembles a key data structure, and some stubs for interaction with Feature's 2 and 4.
Team B starts work on a couple of stories from Feature 2. Applying the 'shoot a bullet through the stack' principle, they decide they need a little bit of UI, a slice of a middleware thing that interacts with some corporate resources and assembles a key data structure, and some stubs for interaction with Features 1 and 5.
Team A decide to create their UI in the simplest possible way consistent with the requirements, and they hand code some HTML and JS that deals with their stories. They make a middleware component that builds their data structure in JSON via API A exposed by a corporate database. To keep it simple, they express their stubs in terms of JSON fields exchanged with Feature's 2 and 4.
Team B decide to create their UI in QT, because they know the company has used this before, they have some QT skills, and it's not inconsistent with the requirements. They build a middleware component that transforms a data structure received from Feature 1 using data received from a corporate database via API A. They create stubs for the interfaces with Feature 1 and 5. To keep it simple, they assume the data from Feature 1 will be delivered as a binary blob.
There are numerous very obvious problems here. You can argue that stubs do not form part of 'done' code, but without a well defined interface to implement and use, what choice is there ? Obviously some decision must be made about how to do the UI for the product as a whole or else it will be an unusable mess. What isn't obvious is that the middleware component which Team A have build a slice of, shares a considerable amount of functionality and dependencies with the different middleware component that Team B have built a slice of. We've just run one sprint, but we are already accumulating technical debt. Not to mention the fact that, as designed, Feature 1 and Feature 2 will simply not be able to communicate.
Of course, this particular train crash is artificial, but let's have a look at what the options are for avoiding it.
Well, perhaps the Product Owners ( there are different product owners for each feature in this company ) could have done something ? However, the Product Owners are not in charge of how things are done, even if they can see the sense in not duplicating effort and not incurring technical debt. Even if the Product Owners work together very closely, they are unlikely to feel that defining the technical detail of interfaces is part of their role - they would be correct.
Perhaps the Scrum Masters could have seen this coming ? Well, yes. Experienced Scrum Masters would probably be shaking their heads before Sprint planning even started. But the Scrum Masters do not own the product, nor do they - acting as Scrum Masters - have any responsibility for the design or technical implementation of the increments. They are not, in general, domain experts. They cannot demand that work on some particular aspect of the Product is done. Pointing out that 'guys, you don't have any architecture' does not magically create it.
Perhaps the Scrum developers themselves could have taken responsibility for avoiding these problems ? It wouldn't take a genius to discover that another team was working on the other end of an interface, or that this team was also creating middleware dealing with related data, or that this team were making different assumptions about UI.
However, for all that intrateam communication is emphasised in Agile, little emphasis is placed on interteam communication, with no formal mechanisms spelt out. That doesn't mean that it doesn't happen or is forbidden. But I've noticed that it is definitely possible for two different Scrum teams working on the same product to not communicate during a Sprint, since the Agile assumption is very much that a team's work is 'contained' or 'independent'. They may even be in different continents, let alone different offices.
Therefore the $100000 ( or substantially more... ) question is this. How and when ( and by whom ) is architectural design created and maintained in an Agile environment ?
I believe the key to understanding this problem is to accept that Product Architecture and Product Code are two different deliverables, albeit with a dependency and a common goal. The goal is to deliver the Product envisaged by the business.
Agile recognises that development of requirements, code, and tests are strands of the overall development problem that naturally run in parallel, and that code and test creation in particular benefit from short iterations.
Architecture is just another of those strands, but it is separated from the code creation problem by virtue of it's scope, which is broader and more abstract, and by the dependency relationship, which implies that relevant Architecture must be delivered before code that makes use of it.
None of this means that Architecture cannot be developed in an Agile manner. Change is as much a challenge for Architecture as it is for Code, and Agile principles are just as relevant.
The more radical Agile viewpoint is that, in fact, there is no such thing as Architecture - there's just a bunch of code that works.
So in this view, the creation of architecture is guided by the problems that emerge in the sorts of situation described above. In effect, architecture is just one of the results of fixing bugs and resolving technical debt in the code. In effect, the architecture supports the code that exists, and only the code that exists.
Therefore starting a complex product must be a slow and messy process, as the basic underlying structure is worked on only when it is missing and has caused a problem. Defining this structure will then have knock-on effects for multiple features, with a lot of rework needed in the early part of the project.
However, because iteration is rapid, and the inspect/adapt principle is constantly being employed, and team members are capable of architectural thinking when required, the product moves in the direction of increased quality and functionality, with only the architecture that is necessary.
I must say, I have never experienced this approach. That means I can't say it doesn't work. Obviously if the project is so small that it can be owned in it's entirety by a single team, this approach feels more natural.
But when there are multiple teams, the one aspect of this approach that leaps out at me is the need for very high bandwidth communication between teams, and particularly between Product Owners. The creation and maintenance of shared structure presents a challenge to the notion of self contained teams working on independent backlog items.
If we accept the idea of treating architectural design as something that precedes the creation of working code, there are a number of ways to go about this.
One technique is to include 'architecture stories' on our product backlog, and ensure when choosing non-architecture stories that all the necessary architecture stories have already been done, by some team somewhen.
Note that we do not have the option to have a 'Feature team' develop the architecture for that Feature - as noted above, this is a contradiction in terms. Architecture stories will be broader in scope, so we will certainly have the situation in which each team works on architecture that other teams will end up depending on. This complicates the Product Owner's life somewhat.
Although using a shared architectural model can help with communication, extensive inter-team communication will be required.
The DoD will be complicated due to the need to accommodate both architecture and code deliverables.
The delivery of business value will appear to be diluted, as architecture per se does not deliver value directly, only the code based on it does.
The problem I have with this technique is that there are several complications and no clear advantage. Coordination of architecture development becomes a key concern for Product Owners, who are using their teams for the purpose. It can be very difficult for Product Owners, who are not normally themselves architects and who will not themselves 'own' the architecture model, to design 'architecture stories' and to know the state of readiness of the architecture with respect to specific user stories. The developers, in effect, have to tell the Product Owners what the architecture stories should be, and this is an inversion of normal information flow in Agile.
Maybe we can de-spaghettify the problem by interleaving architecture development with code development in phases. We spend a sprint or two preparing all the architecture required for the following code phase, and repeat as necessary. This may help with focus, as every team knows that either architecture or code is the thing to be thinking about ( and talking with each other about ) now.
But there is a certain degree of inflexibility here. What happens if not-quite-enough architecture has been defined at the end of an architecture phase to enable full use of the available resource for the code phase ?
The uncomfortable feeling that I have about this approach is that it seems to veer in the direction of traditional iterative development. There's a 'mini waterfall' spanning every few sprints. It limits responsiveness to some extent.
The previous two solutions rely on teams having 'built in' architecture skills. Perhaps there's a dedicated architect in each team, perhaps the team draws on the services of an architect pool, perhaps the devs have the necessary skill set, but the architecture function is conceptually bound to the team.
This solution separates architecture development from code development by running different teams for each purpose.
Product Owners for code development become customers of the Architecture team(s). There is no dilution of focus or switching of modes.
The Architecture team(s) has a Product Backlog which is refined based largely on inputs from customers. The Architecture team ensures that Product Owners for code development can easily discover the state of readiness of the architecture for given use cases and requirements. Architectural change can happen on the same timescale as code change, though code change will lag.
The clear drawback to this approach is that it requires 'architectural resource' as distinct from 'code developer' resource.
So where does this leave us ? I have the following to offer :
1. Architecture will not generally be developed by the same people that create the code. However, this is not a strict rule, it simply represents what is easiest in practice. It is possible to envisage development strategies and team structures that avoid this division of responsibility ( see above ), but they are difficult to manage in a Feature-focused environment due to the scope of architectural design, and run the risk of creating 'mini waterfall' process.
2. Because Architecture spans Features, all Product Owners dealing with Features are stakeholders in Architecture development. They may direct this development collectively as Owners, or it may have it's own Ownership model.
3. Architecture can be developed using Scrum - but the output is not working code, it is well defined Architecture. The customers for this are the Product Owners dealing with Features. The users for this are the Developers creating code for Features.
To further make the case for Agile Architectural development, I'll try to map Scrum artefacts to the architecture domain.
This is not entirely straightforward. The problem is that Scrum expects to have well defined, bounded tasks on the Product Backlog. 'Create some Architecture' is not such a thing.
So, what defines and bounds architecture tasks ? It cannot be Features per se - as explained above, Architecture is concerned with a 'whole system' perspective. A Feature will certainly employ ( depend upon ) particular architectural elements ( components, interfaces, behaviours ), but the definition of those elements will take account of concerns which are not Feature specific.
Requirements, howsoever they are expressed and at whatever level of detail they are captured, ultimately define the product. So the starting point for Architecture is almost always analysis and refinement of requirements. Architectural design decisions should normally be linked to the requirements which have led to them.
So it can be argued that requirements bound architectural design tasks. However, requirements are not defined in architectural terms, there is not a 1:1 mapping between requirements and architectural elements, and there is no guarantee that the requirements at any given point in time are complete and consistent.
One common stepping stone towards architecture and code often emerges from consideration of requirements, and that is the Use Case. Indeed, it's a hardy perennial debate whether Use Cases derive from requirements or vice versa. I will make no assumption about this, save to note that there may be non-functional requirements that do not translate directly into use cases.
I will, however, assume that both requirements and use cases are captured by the architectural model before any actual architecture is created. It is this combination of requirements and use cases that can be used to bound architectural tasks. Note that architecture is no more set in stone than code - change in requirements, for whatever reason, may drive change in architecture with knock on effects on implementation of affected components.
Following on from the argument above, the Product Backlog for architectural development will frequently revolve around use cases. I say 'revolve around' because I don't want to pretend that use cases are the only thing that architecture must concern itself with. The business may have commercial and strategic constraints which also influence the design, e.g. the choice of a specific hardware platform. But for the purpose of defining and ordering architectural work, use cases are important.
I am not arguing that all architecture tasks must have a 1:1 mapping to use cases. This simply won't work, particularly when considering non-functional requirements ( aka Quality Attributes ). What I am arguing is that when defining architecture backlog items, it is useful to work backwards from use cases, so that the question 'do we have the architecture needed for use case X ?' can be answered.
This approach has a key benefit. Because use cases are user focused, they lend themselves to being interpreted as tests - if the user should be able to do this ( perhaps with various sorts of error or exception behaviour ), then a successful implementation of this use case will allow a user to do this.
Note that I've referred to 'implementation' here. This is the link between architecture and code. The logic is straightforward - because sufficient architecture has in theory been prepared to support the use case ( this is known as a result of the approach described above ), then the use case, or perhaps some portion of it, may now be implemented, subject to the usual Product backlog refinement considerations.
In fact, for architecture tasks, the concerns when refining the Product Backlog share a great deal with those for implementation tasks - What is the value of this item for the business ? What technical risk do they entail ?
However, the concerns are not identical. For example, meeting INVEST criteria may present more of a challenge for architecture tasks, as they are inherently broader in scope, and concerned with interactions between different components in the system. The organisation also needs to have a clear vision of what 'testable' means in the context of architecture - this is a deeply thorny topic that I'm going to chicken out of examining here 🙂.
There is also a common issue with architectural work, which is the tendency to conceive of architecture as being a stack of horizontal layers corresponding to different levels of abstraction or refinement. This leads to a thought pattern that wants to complete one layer at a time.
This is dangerous, as it can give rise to a 'big bang' vision of architecture in which nothing is done until it is all done. By emphasising the need for architecture to support use cases, and adopting the same 'shoot a bullet through the stack' paradigm that is useful for code, this tendency can be counteracted.
So what is 'An architecture sprint' ? Does it look any different to an 'implementation' sprint ? Essentially, the answer is no. Or should be no.
The sprint will have a Goal, which may be expressed in terms of enabling certain implementations as that is the ultimate business benefit of architecture. However there's a lot of work buried in architectural decisions, and it may be that the most important decisions at any given time do not translate directly into enabling implementation. This is a decision for the architectural Product Owner together with the team.
What is important is that the job of achieving this goal, and the execution of the tasks deemed necessary to do so, is treated as a shared responsibility.
The reason I emphasise this here is that it is very tempting when dealing with architecture to make particular individuals responsible for different parts of it. Those individuals become Experts, and they are automatically expected to deal with 'their part' of the problem. If you do that, you end up with an Architecture Department in miniature, with every person working independently, not a Scrum team.
No matter how high-functioning these individuals are, there is value in ensuring that they make commitments, share their problems and insights, and develop solutions jointly. This team perspective is fundamental to Agile and to Scrum, and losing sight of it will render the whole apparatus of Scrum ineffective.
An increment is achieved whenever the DoD has been met. Which just boots the problem down the road 🙂. What does a DoD look like for an 'architecture increment' ? What does quality mean when talking about architecture ?
What this amounts to, given that I've already said that the deliverable for Agile purposes is 'well defined architecture', is to interpret 'well defined'. Well defined cannot mean 'complete' in the sense of 'the entire system', or else we are back to big bang, waterfall thinking.
Architecture will usually be created in the form of a 'model', this being an expression of the architecture using some formal notation, possibly UML, possibly SysML, although there are other options. It is possible, with appropriate tool support, to automate various analyses of this model. So it may be possible to refer to the results of such analysis for the purposes of DoD. Maybe a simple analysis could check naming conventions, missing content in various fields, etc. This is unlikely to be all that the team want to check, but just as with code deliverables, automating as much QA as possible and running those tests frequently is important.
Two key things that must be known before an architecture increment is done are 'what new use cases are enabled ?' and 'what existing use cases are affected ?'. Traceability from architectural elements to use cases is something that it may be possible to test automatically. Certainly there should be no new architectural elements which cannot be traced to requirements.
Review may well form a part of architectural DoD, but it's important that review does not become a 'big bang' thing that only happens at the end of a sprint - frequent small cross-checks with other team members is a better model for review in this context, provided that the execution of such review can be demonstrated for DoD purposes.
Personally, I believe it's possible to develop Architecture in a way that is consistent with Agile principles, and integrates reasonably neatly with development of working code. Architecture only translates into business value when - or if - it is used to guide the creation of working code. There are some pitfalls and unique problems, as noted above, but they are not insurmountable provided that both the developers ( architects ) and the organisation have a clear vision of where Architecture as a deliverable fits in to the overall Product creation process.
© Mark de Roussier 2022, all rights reserved.