This is the first in a series of articles which deal with the values in the Agile manifesto. “Individuals and interactions over processes and tools“. In agile, we acknowledge that the stuff on the right (the stuff in italic) is important. But we value the stuff on the left more (the stuff in bold). This article will unpack this particular value – it will ask several questions:
• What does this value bring to my data warehouse?
• How can I apply this value in my data warehouse?
• What mindset change do I need to go through to adopt the value in my data warehouse?
Before we start, we need to remember that in a classic scrum setting, the roles have been flattened. So our analyst is our developer is our tester, our scrum master is our creative genius who “makes a plan” and our product owner owns our product backlog. We also need to remember that our customer has far greater input into the end product. It is not uncommon for the product owner to change the terms of engagement halfway through delivery – this is what makes agile so great. So for example, a customer could say “wow this is a cool button, but can it do this?”. Even though this has not been specified or documented anywhere, suddenly this may become the highest priority for delivery. THE RULES OF THE GAME CHANGE. Deal with it. Live with it. Accept it. Or go home…
Similarities to rugby
So in rugby, very often the play of the day is something not delivered in the script. As a Springboks supporter, I can tell you of some magic moments. Joel Stransky lining up for a drop kick that would bring us the world cup. Ricky January chipping the All Blacks and winning the game as well as the Tri nations. Not so memorable was also not in the script. Larkham drop kicking the Aussies to victory. Savea fending off several defenders and side stepping his way to a tri nations victory. The point is that the particular play was never in the script. It was never documented. It was never coached. Yet it was that one particular play which brought the most value to the game.
In agile scrum, you will find very often that the stuff you don’t plan for actually brings the most value. So on my current project, we have been breaking our backs getting stuff in using a scrum methodology. We have been putting together sprints, monitoring burn down, doing retrospectives. Yet, actually the most value is in what our customer is going to do with the data. There is enterprise value in that. It certainly is very unknown right now – it hasn’t featured on any product backlog, it hasn’t even been whispered about in sprints. But it is real and it is upon us. Very much the same as rugby where you may have trained the whole week on defense but maybe the outside centre has some magic you didn’t train for. At match day, there is no way for you to say “hey outside centre, you aren’t allowed to do that because we are not prepared”. It is match day. You adapt. You improvise.
In scrum, the individuals who can master the art of improvisation over the art of perfecting a process or a tool are the individuals I want in my team. I still need the guys who respect the process and are kings at the tools, but I need even more the guys who don’t mind the rug being pulled out from under their feet being replaced by wooden floors…
Individuals and interactions over processes and tools
I’m sure that you know the old story about bacon and eggs. For our wonderful breakfast, we can say that the chicken was involved, but the pig was committed! In our project, we need to split out the chickens from the pigs. The pigs are the guys that are invested in the sprint. The chickens are the guys who have a say in what goes on, but are not invested. Why is this important? Because for your sprint to work and for you to achieve a goal, you can only really count on the guys that are invested. So who are the pigs? Simple – the product owner, the scrum master and the development team. Remember, the development team is flat – there is no “fearless and mighty leader”, it governs itself. The development team would consist of analysts, developers and testers. It is not uncommon for a developer to work with the product owner or the consumer to come up with some product backlog item. It is also not uncommon for a product owner to turn around and then say “wow this credit risk report is great but I need the ability to slice and dice by department”. In such a case, it is fantastic that you know how to jockey the tool used to load or the tool used to report. But what is more important is the interaction between the developer and the consumer in refining the requirement and producing a better quality product.
So in our case, we already have user stories which deal with landing data and reporting from it. We are not quite where we need to be with velocity and our burn down, but we are getting better. We have a data warehouse which is still a small baby, but it now has just gotten it’s first tooth – not crawling but at least eating solid foods. We need to look at some ways our end consumers are reporting on our data (dealt with in other user stories). The only way we can do that is through individuals (a developer and a consumer) and interactions (various sessions and discussions on usage of data). We could have said that “we need to wait for a data model”. We could have said that “we need to wait for a requirements document”. In essence, we could have placed the process or the tool at a premium. It would have taken us a while to get there (rough estimate of 6 months). Instead, we place the individual and interaction first.
So get the dialogue going. DO NOT DISCUSS THE DESIGN, but rather discuss the usage. Those outputs in turn feed into the processes and tools (so build me a model with x entities and attributes and y slicers and dicers, and while we wait do z with the data to satisfy the immediate requirement – acknowledge as technical debt and create a backlog item to deal with it).
“Doing agile” versus “being agile”
So you have a standup and Jira? I am so happy for you…. But you simply cannot meet the end user halfway – you tell him it will take six months for him to get where he needs to go. Would it take us six months? Definitely not. We would know almost immediately which entities and attributes we are touching, we would know which ETL/ETL processes are affected and which reporting requirements we do have. The difference is that we are making a difference six months earlier than we would have made if the inverse had applied. In fact, in those six months, I expect the requirement to change several times. “Hey, you are slicing by department, but how about excluding managers” or alternatively “this report is great for managers but how about an operational report for employees”
Vox Politico
Traditionally, developers would look at the spec. If it isn’t in the spec, they have a chance to bail out. In my opinion, this is a cop out – at best. If you as a developer cannot even work out that you need to slice and dice by department and that there is a second requirement for an operational report for employees, then you are at best a jockey for the tool. And that isn’t even agile, it is just common sense. But politics do come into play in waterfall. Rework is frowned upon, in specific late in the development lifecycle. What makes this so beautiful is that the interaction between the individuals leads to some amazing stuff. In our case, the stuff we are doing that isn’t really documented in a backlog or sprint will drive insane business value.
In my next blog I will be discussing the next value – “working software over comprehensive documentation”.
I would love to hear from you, please feel free to comment.
Doc Phil