Thoughts on DevOps - Organizational Structure and Culture
In the previous post I talked about bottlenecks in organizations, with a view from the DevOps transformation perspective. This article will focus on the close relation between the organizational structure and the culture.
Table of Contents
The point of any organization is to deliver a product or a service (further called Product) to the customer, partner, patient, etc. (generically called Customer). It doesn’t matter if it’s an NGO, a private business or a state agency, the only important thing is making sure that the full attention is kept on delivering the (right) Product to the Customer. Since without the Product (and the Customer receiving the Product) the organization doesn’t have a reason to exist, the way we set up the organization needs to reflect this.
Looking at the existing structures, we can easily notice that the different teams have competing or even opposite targets from the Product’s point of view. Business will concentrate primarily on acquiring new Customers or offering current ones more of the Product, Development on new features, Testing on finding the problems and Production on stability.
While some of these targets might intersect, others are in direct contradiction with each other. Take Development and Production/Operations for example. Obviously, you want to have as many new things as possible for the Customer (from new or better features for the current Product to completely new products). But you also want to be able to offer the Product without disruption. Every change is prone to break something, so the team in charge with the stability is extra
Numerous times I’ve heard comments like these:
- “The developers always break things.”
- “The IT Ops guys are blocking rolling out my change again.”
- “Who thought to enable this feature in this way? This is stupid, it will break production.”
- “QA still hasn’t tested my release. What are they doing there?”
Every one of these quotes shows multiple phenomena in full effect:
- Shifting blame
- Them vs. us
- Lack of understanding or interest in the work of others
Structure generates culture
When looking at the impact on the Product, one cannot ignore the close relation between the organizational structure and the culture it promotes.
Blameless culture originated in the healthcare and avionics industries where mistakes can be fatal. These industries nurture an environment where every “mistake” is seen as an opportunity to strengthen the system.
Dealing constructively with failure is one of the key requirements for a successful DevOps transformation. Yet, adopting it becomes exponentially more difficult if your organizational structures and processes don’t allow, support, or encourage direct communication. Instead of seeing problems or incidents as chances to improve, people focus on whose responsibility it was to prevent the disruption.
A lot of new concepts (and sometimes old concepts with a new name) started appearing over the last few years, in connection to DevOps. From Blameless Culture and Lessons Learned to Self Organizing Teams and Fail Fast. Always try to understand what a certain concept means, what it means for your organization and, more importantly, for the Product, and always be prepared to explain it to others.
For establishing a DevOps culture, the whole organization has to work together, using the same core values, speaking the same language and having a similar understanding of the vision. This cannot happen without one or more actors insuring that the transformation is on the right path.
I like defining “DevOps core values” those cultural traits, without which DevOps cannot exist:
- Trust - equally important within a team (the manager has to trust the team) and across the organization (one team has to trust another when doing something)
- Transparency - the difference between “The Infrastructure Team doesn’t do anything the whole day.” and “I’ve heard the Infrastructure Team is working all hands on deck on a top priority project, we’ll have to adapt and work on another task now.”
- Feedback - short and quick feedback loops are important in order to ensure transparency and trust (and let’s not forget, monitoring and measurement are forms of Feedback)
- Blame Free - also see the example used in Blameless Culture
Underlying these core values are Change and Communication. Every step of the way there can be an opportunity to improve the Customer experience, to improve the Product or the processes. Only an organization willing to adapt (and have an open communication to the employees about it) can manage to take the steps towards adopting DevOps.
There are obviously more values to DevOps than the ones listed here (such as taking responsibility, knowledge-sharing, team-enablement etc), but these build upon the “DevOps core values” mentioned above.
Considering that DevOps is seen as just a buzzword by many, with 9 out of 10 people having a different understanding about it, aligning the language used becomes both important and urgent. They need to speak about the same thing when they are talking not only about DevOps, but also about the core values. One way to manage this is by setting up workshops, designing exercises where people from the entire organization, regardless of their role, intermingle. This way, they can learn what it’s meant by Transparency or Change for example, in the specific setting of their day-to-day work.
The same as with the Common language, having a clear Vision of the target isn’t enough. Communicating the Vision, making sure that every person understands it and how it will impact their work is one of the cornerstones of a successful DevOps transformation.
Tying together the core values, common language and the vision, establishing a DevOps culture cannot happen without Coordination. The same core values already mentioned have to be applied to the organized transformation. It’s not enough to merge Development and Operations teams and hope for the best. Someone, ideally in a high position within the organization, has to make sure that people don’t fall back to old and known patterns.
I’m going to give you an example here:
I used to work with a team that was established as one of the first DevOps teams within a very large organization. This was a team with a lot of Systems Engineers and Application Developers. Due to the size of the team (~40 people), it was split into three smaller groups.
Normally, considering the focus on the Product, one would want to split the big team into smaller products. Each of the three groups would be a self-supporting unit, that can work independently from one another, with clear communication interfaces (both for the teams and the smaller products). Unfortunately, due to the lack of coordination, an unclear vision and different understandings of DevOps, what happened was the creation of two development groups and one operations group. Effectively, they have recreated the organizational silos within their smaller group.
“I’ve had a retrospective with the team.” someone told me. Surprised (how do you hold a retrospective for 40 people across three countries?), I asked “The entire team?”. “Yes!” came the answer, “And they were complaining about the operations guys.”. Turns out, the only ones present were the developers from that location. Even though it was communicated “We are all one team!”, the people didn’t really see it that way. They were in a functional group that segregated them from their colleagues based on what they were doing instead of what they were working on.
When starting a DevOps transformation, one has to keep the focus on the Product and the Customer. Achieving synergy between the structure and the culture can be the deciding factor in a successful DevOps adoption.
When I started to write this article, I did not imagine it will take so long or need so many rewrites. In the next article I’ll be taking a deep dive in the structure of a DevOps organization.