The Elephants in the Agile Room

Introduction

Agile software development has many advantages and offers many benefits. As in life and business where every benefit comes at a cost or sacrifice, Agile too has limitations and comes at a cost. More importantly, it is easy for organizations and teams new to Agile development to stumble into pitfalls and lose their enthusiasm for Agile. As someone once quipped “there are a few elephants in the Agile room” and this article attempts to highlight some of the major limitations and pitfalls and suggest ways to overcome or avoid them.

• The Agile Manifesto – Theory vs. Practice

The Agile Manifesto (www.theagilemanifesto.com) states 4 values and 12 principles that embody all Agile approaches. Earlier articles of this blog have covered the values and principles in detail and this article will not repeat that. Essentially, Agile revolves around three core concepts – Transparency, Inspection and Adaptation. While they are great concepts in theory and make a lot of sense as applied to Software Development, many organizations struggle with them in practice.

The biggest hurdle to transparency, in practice, is organizational politics. Unfortunately, Scrum teams, Line / Functional Managers and Business stakeholders continue to erect walls that hamper transparency.

The biggest hurdle to the concept of “inspection”, in practice, is the socio-cultural orientation in organizations towards failure. Given that the basic idea of inspection is to fail fast to win fast by ensuring processes are running fine, most organizations tend to avoid failure thereby prolonging the completion of backlog items, in the long run. They tend to continue running flawed processes in the flawed view that if something is running then it is not broken.

The biggest hurdle to the concept of “adaptation”, in practice, is the difficulty in adapting many powerful stakeholders outside the Scrum team to the right processes and practices. In practice the adaptive decisions made are often by the loudest and most powerful stakeholder and rarely by the Scrum team.

A big limitation of Agile, in practice, is the assumption that all stakeholders in the organization are also as agile as the Scrum teams. Business agility is a great notion in theory and is often swamped by organizational inertia.

• Framework flexibility – strength as well as liability

In other articles, we have seen how Agile is a guiding framework of values, principles, and practices. It is not a method as is commonly and incorrectly referred to. Other than the Agile Manifesto, there is no universal standard or “official” guidelines for how Agile should be adopted and practiced. Coaches, Consultants and Practitioners often evolve their own “ways” of applying Agile concepts. It is not about applying the concepts as much as it is about “being Agile”. There is also no official body or organization that is entrusted with owning and governing the framework, simply because Agile is not one approach but an umbrella term for several different but related approaches such as Scrum, Kanban, Extreme Programming, Lean Software Development, etc. While the Agile Framework offers the flexibility for teams to “adapt” practices to their organizational context, the lack of a standardised approach acts as a limitation. This is particularly relevant to organizations that have just adopted Agile. Usually, organizations new to Agile engage an internal champion or external coach to get them started on the Agile journey. As such every organization’s Agile journey evolves in a different direction based on who is guiding them. Some organizations fall into the trap of blending Agile practices with their current or past development approaches such as Waterfall. This results in the new initiative becoming ScrumWaterFall rather than Scrum. It is best for organizations to adopt the pure framework of Scrum and adapt to it rather than bend Scrum to fit their environment.

• Dependence on the Agile mindset

The effectiveness of Agile approaches is heavily dependent on the right mindset within individuals and the team. The prevailing culture within the software development department of an organization predicates the successful adoption of Agile. Many organizations whose core business is not Information Technology but something else such as Retail, Manufacturing, Financial Services, Health, Transportation, Government, etc. have different organization cultures that are unique to their industry in general and to their company. Usually the company culture will play a dominant part in shaping the culture of IT departments that undertake software development. It is very difficult then for the software development teams to develop their own culture that aligns with Agile behaviours. Thus, organizational and department-specific cultures clash with the required Agile mindset and cause conflicting situations. Some real-world examples include;

• Setting up of hierarchical structures in Agile teams because that (control and command) is the company culture

• Enforcing “Project status reporting” for Agile projects because that is company policy (under the mistaken notion of strong Governance)

• Engaging vendor / partner in Agile projects on fixed price terms to minimise risk as that is how the company operates (Strong vendor management)

• Critical dependence on Product Owner for success

Although Agile (Scrum specifically) aims to distribute responsibilities across the team, it is commonly accepted that the Product Owner is the lynchpin for the Scrum team. The product vision and decomposition of the product into Backlog is heavily dependent on one person, the Product Owner. As such, the Product Owner becomes the single point of failure for the entire Scrum team. There is not much the Development team can do if the Product Owner is unable to articulate the vision for the Product and help breakdown the product into its component and features. That responsibility then falls on team members with business analysis skills and product knowledge. However, they lack the authority that a Product Owner has. In traditional approaches to software development, a vision for the product and its requirements are usually formulated during the requirements gathering phase by tapping into several business users and stakeholders. Therefore, there is no single point of dependency as in Agile.

• Degree of preparedness required before starting sprints

This follows from the earlier observation regarding lack of more detailed guidelines for the Agile framework. Many organizations and Scrum teams struggle with the level of preparedness required prior to starting the first sprint. Typically, a project is launched followed by the formation of the Scrum team and after the customary Scrum planning meeting the team starts the first sprint in a high degree of excitement. After the first couple of sprints it becomes obvious that they are still doing planning work in the sprint instead of developing software in an iterative manner. In essence, the Sprint team is working on backlog items that are just not sprint ready. This results in teams working on the same User stories, sprint after sprint without being able to complete them.

It is essential that an Agile project goes through a proper initiation stage before proceeding to the execution stage (Sprints). However, there is no formal or standard guidance for how Agile teams should manage this as part of the framework and it is left to individual teams to do it whichever way suits them best.

Recommendations

1. Implement regular and periodic “Organizational-wide Retrospectives” to highlight what’s working and what’s not. The power and influence of social proof is phenomenal. It will be amazing to see how quickly rogue teams will start using Agile practices the proper way when they see other teams doing so and getting good results.

2. Engage a great Agile coach to design and implement proper Agile practices and nominate a high-level stakeholder to continuously champion Agility, long after the Agile Coach has gone. Like any initiative will flounder without a champion so too will Agile if left to itself. Many organizations and individuals tend to gravitate towards the mean over time. It is important to introduce course correction every year.

3. Engage proven organizational change experts to introduce behavioural changes for business agility, outside traditional IT departments. Changes in the business community almost always drives changes in IT Departments (not the other way around).

4. Involve all key stakeholders in the Agile journey and conduct immersive and regular workshops around the Agile Manifesto.

5. Start with the Business, not with IT Departments. Develop great Product Ownership capability within the Business community. IT cannot and should not own the Software products that will be used by the business. This is as silly as the Auto company that produces the Car and the Mechanic that services it, taking ownership of the car you drive every day.

6. Treat the Agile initiative in your organization as a journey rather than as any destination (There is no end-point to being Agile. It is a way of software development).