The science behind Agile. Research and theories to demonstrate that the approach really does make sense

The worst thing Frederick Winslow Taylor ever did to us all was to call his management style scientific. After all, his main maxim, “order and control”, was only a description of a certain status quo that had persisted for centuries (and, as we already know well today, just because something is done the way it has always been done doesn’t mean it’s done right). Many years had to pass and a lot of research had to be done before we could finally say that science, in fact, backs Agile, a methodology that boosts creativity and performance even as it hands over much of the “power” to “ordinary” employees.

The science behind Agile - research and theories to demonstrate that the approach really does make sense

Goodbye, Mr Taylor, or the twilight of “scientific” management

To cut a long story short, Taylor’s theory of scientific management simply comes down to top-down planning.

Whoever is up there on top, the theory goes, knows best and should thus organise work for those lower down in the organisation (so that power descends vertically).

Like machines, those on the bottom are supposed to limit themselves to carrying out orders.

That said, of course, since they have the power and the knowledge, those on top get paid a lot of money. Those on the bottom who, in theory, have little knowledge and responsibility, get paid less.

However, research shows that in this paradigm, low-level employees not only have little money but also precious little motivation to work effectively.

On the face of it, everything seemed to work fine in Taylorism: people at the bottom of the ladder worked like smooth conveyor belts and everything was broken up into sequences.

Reductionism was the rule and, of course, it was a form of scientific thinking. But too much reductionism also meant that an employee was likened to a machine. And, as we know, there are quite a few important differences between people and machines.

For instance, human performance is not linear. We don’t work the same in the second hour as in the sixth, tenth or twelfth. This is why, in the 1940s, people began to question Taylorism and break away from it.

Agile didn’t come from nowhere. The scientific basis for agile management

The agile mindset, or the agile approach to project and organisation management (or the management of any process, for that matter) rests on robust scientific foundations derived from a range of different fields.

The interdisciplinary evidence that supports the validity of the agile paradigm encompasses research in psychology, economics, systems theory and many other disciplines.

Further on, we will discuss some such examples to see how agile is indeed backed by science.

Long live the king, or the theory of complex systems: The theory of complex systems and, in particular, the work of Ludwig van Bertalanffy in general systems theory played a key role in the development of Agile. The theory is essential for understanding how complex interactions within organisations and projects impact their effectiveness and odds of success.

Even though it seems obvious to us today that there are interrelationships to be found on every corner, people long believed that various functions could be isolated and looked at separately.

This was also applied to various functions in an organisation and this is why siloing has survived until now as one of the worst things that put the brakes on many enterprises.

One department works out its standards, KPIs and other metrics and never stops to think whether putting them into practice will have an impact on what happens later globally.

In an agile framework, while we might work iteratively, we still need to be able to see the bigger picture. We need to understand the relationships between what we are doing today and what has already been done; more generally, we also need to consider how our actions today may impact what happens in the nearest future.

In contrast, the classic waterfall (cascade) methodology is more akin to what Taylor had in mind: one step is completed and, from the perspective of those who complete it, the work has been done.

Now, let others worry what to do.

Naukowe podstawy Agile 2 ENG

However, as emphasised by Bertalanffy’s theory of complex systems, it doesn’t matter if we design highly complex IT systems or manufacture cars: we always interact with reality.

Taylor’s paradigm chooses to ignore reality altogether, while Bertalanffy’s always takes stock of the context.

For instance, say we want to develop an IT system for car insurance management. We need to understand that the insurance industry is a very complex ecosystem; we cannot ignore broader industry needs, its vocabulary, its directions of development or customer expectations.

Psychology and Agile. Kahneman and Tversky

In our quest for scientific evidence, let us now enter the realm of psychology. We will start from a famous study by Daniel Kahneman and Amos Tversky (1979): “Prospect Theory: An Analysis of Decision under Risk”. 

Kahneman is the first psychologist to win the Nobel Prize …in economics. His research focused on decision-making under uncertainty (the cognitive errors we make and the ways in which we navigate a reality that is unpredictable and variable) and showed why quick iterations and regular reviews are more effective at ensuring adaptability to change than careful long-term planning.

When we put our brain in a situation where it needs to take many decisions over a short period of time, we significantly increase the likelihood of error. And let’s not forget that in IT development, a decision needs to be taken every time we construct a requirement. So the best strategy would be to take decisions in small chunks and quickly check to see if the outcome is indeed what we expected.

Kahneman and Teversky’s study shows that some of our decision-making tendencies may limit our capabilities. A blind faith that we can predict everything and always work out the best solution is not just very naive but also rather risky.

If you have ever worked in projects before, you probably understand how many reverse loops you need to go through where you need to change something or start over from scratch and how much money is sunk into such patch-ups.

Scientific management. Gantt chart

One of scientific management’s pet artifacts is the Gantt chart, even though the evidence for its effectiveness is very controversial. There is abundant research, conducted on more than 500,000 Gantt charts created in projects between the 1950s and the 1990s, which shows that c. 98% were radically revised at least three times.

This demonstrates that, firstly, initial predictions were ineffective and, secondly, someone had to do the same job several times before it finally worked.

Decision overload

Kahneman clearly demonstrates that planning things over time and taking decisions are not our strongest assets. This is why it is important to have a vision of the larger goal we are trying to achieve and split our decision-making into smaller chunks.

This is further reinforced by research from the past two years; recent studies show that there is a limit to the number of decisions that an individual can take in one day (sleep is assumed to somehow reset this “decision counter”).

Even though it is otherwise a quite precise instrument, our brain treats each decision separately. It doesn’t matter if it needs to choose between coffee and tea or decide whether we should buy a company worth a billion dollars.

Each decision, no matter how trivial, puts the same load on the brain. There is even an anecdote making the rounds in organisations: highly effective individuals have 15 identical shirts in their cupboard so as not to overload their brains with constant decision-making.

Even more psychology. Amabile and Kramer

Let’s go on to more recent research: “The Progress Principle” by Teresa Amabile and Steven J. Kramer (2011), which analyses the impact of everyday progress on employees’ emotional states, motivation and perception.

Based on journals, Amabile and Kramer’s research shows how small daily successes and signs of recognition can spark a positive mindset and boost performance. Accordingly, managers should strive to provide support and create environments that positively respond to employee progress.

Transparent communication, regular feedback and the appreciation of effort are essential for effective project management. Just being aware of feedback as a phenomenon (knowing that someone out there is interested in whether or not we complete a task) boosts our cognitive flexibility (we begin to search for more creative solutions) and has a positive impact on our motivation.

This research is very important, or should be, especially for anyone in a managerial position. It encourages us to go off the path of Taylorism, where tasks are simply delegated to others and no news is good news, so that we only get back to our employee with criticism and comments. This approach is a perfect recipe if we want to stifle the creativity and performance of the people we work with.

Even on a biochemical level, very roughly put, feedback (even negative) activates the reward system in our brain. In contrast, just paying our employees a visit and saying “you’re doing great, I’m happy to work with you” fails to activate that same system if it is not accompanied by a reason why.

In sum, whenever we give feedback, we always need to rely on facts.

An average person likes to know why exactly they are being praised.

Psychology again. Martin Hilbert

Another insight from psychology research is a study entitled “Cognitive Flexibility and Adaptive Decision-Making” (Martin Hilbert, 2012), which tells us how we can fire up our cognitive abilities. Such as, for instance, analysis, which plays an incredibly crucial role in software development

Once again, there is ample evidence that quick feedback is of the essence in agile projects. More than that: research shows that quick feedback helps significantly decrease uncertainty-related stress. After all, change is not something our organisms really like. Any change involves that famous “stepping outside your comfort zone”.

If we get regular feedback, we are able to handle change with greater calm. This is yet further proof that short iterations and feedback loops can improve the cognitive flexibility of our employees, who will be able to find solutions faster and quickly deliver whatever is needed in response to change.

When we develop IT systems, especially those suspended in a legal context of some sort, such as is the case in the financial sector, where laws often change in quick and largely unpredictable ways, sometimes the pivots we need to make are really rapid. The sooner our people get over the shock, the sooner they can deliver high-quality software.

Agile and multiculturalism. Moe, Dingsøyr, Dybå

To anticipate the charge that our article focuses exclusively on biosocial sciences, let us emphasise that more technical sciences, such as software engineering, have also studied the impact of agile methodologies on work effectiveness. One such study is “Agile Practices in Global Software Engineering — A Systematic Map (Niels B. Moe, Torgeir Dingsøyr, Tore Dybå, 2010). 

In this case, the authors focused on agile management in remote teams burdened with the variable of multiculturalism.

Here, too, agile methodologies were shown to be superior and more effective at inducing the expected level of performance, despite the reality of multiculturalism. The system of “order and control” (Taylorism) is hampered by many variables that stem from the culture that each of us has grown up in. There are different cultural models: individualistic cultures, such as the Anglo-European culture, but also collectivist cultures, which span large swathes of Asia. This shapes completely distinct visions of authority and power.

As we said at the beginning, Taylorism involves very rigid divisions and clearly defines who is there to take decisions, who wields power, and who enjoys certain permissions. Today, as we work in increasingly multicultural environments, we no longer need to study different nuances: we can accelerate assimilation in projects just by adopting agile methodologies.

Homeostasis, or Ross Ashby

It is now time to go back to the archaeology of research, or the dawn of cybernetics and a monograph entitled “An Introduction to Cybernetics” by W. Ross Ashby (1956). 

Ashby’s book is one of the foundations of cybernetics, which studies the systems of regulation and communication in animals and machines. Agile relies on the principles of cybernetics to understand how teams and projects can remain stable and effective under dynamic conditions. Ashby helps us understand how to manage complex systems effectively through decentralisation and adaptation.

Homeostasis: In cybernetics, Ashby introduced an important concept of homeostasis, which stands for the ability of a system to maintain a constant internal state under variable external conditions. The concept underwrites our complete understanding of how systems self-regulate and adapt.

Cybernetics introduces the concept of homeostasis; it was shown that systems (such as, e.g. project teams, but also things we create, e.g. IT systems) strive to maintain stable internal conditions. On this view, any development involves a transition between two successive states of equilibrium. This means that homeostasis must be understood (acknowledged) and then violated so as to be restored again at a higher level. 

Law of Requiste Variety: Known as "Ashby's Law", it says that the effectiveness of control is limited by the number of available controller states. In other words, a controller must have at least as many states as the system it controls.

Agile applies Ashby’s law in the following way: some power, including formal power, needs to be devolved to teams. If we wanted to manage teams like before (trying to be a Taylorian manager), we would need to be able to control everything that our teams do. An approach of this kind always gets mired in problems, such as micromanagement. By handing over some power to self-organising teams, Agile allows the controller, or our manager, to make sure they have enough possible states to carry out their duties.

Knowledge management theory. Nonaka and Takeuchi

We cannot really talk about Agile without mentioning two special gentlemen from Japan. So let us look at the Knowledge Management Theory proposed by Nonaka and Takeuchi. These two authors conducted several important studies and published several research papers on knowledge management.

Again, in the Taylorian paradigm, knowledge is very unevenly distributed. What makes agile frameworks superior is that, when well-managed, they stimulate not only knowledge transfer, but also knowledge production.

In Agile, knowledge management is not just about sharing information, but also building an environment in which knowledge can be freely produced, shared, and used for the benefit of the organisation as a whole.

Well-managed agile projects are the incubators of knowledge, which then becomes the company’s capital asset. If our management method ignores the processes of learning (i.e. if we fail to manage them or create relevant support mechanisms), we are not agile at all.

We are just pretending.

In this way, we allow capital to leak out of our organisation. If you want to know more about this phenomenon, follow this link: Inefficient software development. Is the learning process to blame?

No agility without innovation. Christensen

To wrap up, let’s briefly discuss “Innovation Theory” by Clayton Christensen. Research into disruptive innovation also offers essential insights into Agile. Christensen observed that companies often go out of business simply because they fail to adapt to market innovation. 

In our context, this underscores the need to constantly adapt to changes and adopt innovative approaches. With its flexibility and adaptability, Agile allows companies to quickly react to new market and technology developments, minimising the risk of overlooking key innovations. 

In addition, Agile promotes a culture of constant experimentation and learning, which is a crucial prerequisite for identifying and tapping innovative opportunities. Thanks to agile methodologies, organisations are able to remain competitive through a constant effort to enhance their products and services. Christensen emphasises that companies need to be ready to redefine their markets and business models when needed, which is in keeping with Agile’s flexible approach to project management. 

Simply put, Agile provides the structures and processes that fuel innovation.

Conclusions

As you can see, Agile is much more than just a project management methodology. It is an entire philosophy of action that rests on robust scientific foundations and promotes adaptation, flexibility and constant improvement.  

So go forth and…agilise, and rest assured that what you do really makes sense. You have solid science on your side?