A senior manager in the operations area of my organisation recently commented:

There is no such thing as agile infrastructure

That got me thinking. I can imagine that adopting an agile approach to infrastructure might be inappropriate in certain circumstances, for example military or medical domains.

On the other hand my current programme needed a completely new infrastructure stack and I found a considerable amount of agility was possible through that exercise. To my way of (probably simplistic) thinking any changes needed from the infrastructure are to provide either more capability or faster capability. With that in mind I’ve found it is possible to architect for agility and deliver the infrastructure in an incremental and iterative manner.

Speculative Agile Manifesto for Infrastructure

The first question is … what does agile infrastructure mean?

The Agile Manifesto seems a good place to start. Can this apply to infrastructure? I believe that, with a couple of word swaps, it can. Here’s how it might look:

We are uncovering better ways of building
infrastructure by doing it and helping others do it.
Through this work we have come to value:

Individuals and interactions over processes and tools
Working infrastructure over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more.

Looks alright to me. I changed three words. "Developing software" became "building infrastructure"; I admit I’m less than happy with "building" but I couldn’t think of a better term. I also changed "Working software" to "Working infrastructure".

Individuals and interactions over processes and tools

We need processes and we need tools but, as with software development, I’ve found that people and relationships are far more important when getting things done on the infrastructure.

Working infrastructure over comprehensive documentation

This is a biggie for me. It doesn’t matter how beautiful the architecture / network diagram is if the system (hardware, operating system, web servers, application servers, application platform, etc) don’t work. But, given that it works, it is better to have the infrastructure documented than not.

Customer collaboration over contract negotiation

This might be less important in the context of infrastructure. Customer representatives can get excited about the user interface of the application, hence we’ll get big rewards from collaboration, but I’ve not seen many customers get excited about the number and type of servers.

Having said that a good relationships between the user group and the infrastructure team, and between the infrastructure team with the vendors, is vital.

Responding to change over following a plan

This is the other biggie for me. As I mentioned above the changes I need from the infrastructure are either more capability or faster capability. I’ve found it possible to get the flexibility I need by architecting for agility and delivering the infrastructure in an incremental and iterative manner.

Architecting for Agility

I believe in having some upfront design work on agile projects. You don’t have to go crazy and I typically allow a month for the software team to get the high level prioritised requirements, release plan, high level software architecture and high level user experience.

For my current programme we also had to put in a completely new infrastructure stack. Actually we were replacing a very small existing stack from a previous prototype. As you’d expect the infrastructure team put together a architecture design. There were several aspects of this architecture that gave me considerable flexibility as the programme progressed. There were two things in particular that gave me flexibility.

• Virtualisation
• Horizontal scaling

All of the environments in my team’s pipeline to live (development, integration, testing, staging) are virtual. So if we need another environment for any reason, e.g. a short term hot fix environment or an environment to test a patch to the underlying platform, it is trivial for the support guys to provide this.

In hindsight I wish we’d virtualised the live environment as well, but it is, in fact, physical. None-the-less I still get some flexibility by ensuring that web servers and app servers can easily be added = horizontal scaling.

With the architecture came a plan for iterative and incremental delivery.

Iterative and Incremental Delivery

From fairly early on we had a clear idea of how our user base was going to increase over time; it was a rise over two years. So although our architecture design showed the end-game, we didn’t need all of the infrastructure from day one. So we opted for a phased delivery:

1. Prototype stack with Sharepoint 2007
2. Interim stack with Sharepoint 2010
3. First half final stack
4. Dedicated search boxes (if necessary)
5. Dedicated integration boxes (if necessary)
6. Second half final stack (including repurposing boxes from interim stack)

This approach offered several advantages:

• helped with on-the-job training up our support guys (they got lots of practice)
• gave us a viable fall back position as each increment
• gave us options about dedicated hardware – we could see whether our infrastructure needed it before investing the money

Other Agile Infrastructure Options

They weren’t necessary and/or possible for my programme, but there are two other options that can offer the same sort of flexibility for infrastructure:

• Content Distribution Networks (CDN)
• Cloud Computing

Other teams in my organisation use Content Distribution Networks (CDN) to provide headroom. The normal in-house infrastructure is used except in cases of extreme load, in which case the site cuts over to a CDN. For me this is just another way to architect for agility.

Hosting your system in the cloud is like the ultimate virtualisation. You want more capability so you just buy more and it is live immediately. Perfect.

Conclusion

None of what I described above is new, radical, or different. But it is Agile in that I could respond quickly to changing circumstances. Virtualisation, horizontal scaling, content distribution networks and cloud computing all provide agility if architected in.

Our programme also demonstrates that infrastructure doesn’t have to be big bang as we are successfully using an incremental and iterative approach to delivery. Another stock standard of the Agile community and a facet of my approach to agile programme management.

References

Agile Manifesto