Do Business Simulations work? The Evidence Case Pt 1

In a previous article I discussed the most common dilemma in business education, that of ‘external validity’ and the need to prove that investment in training (of any type) requires justification on a typically quantitative level. Most usually that it impacts the organisation positively. In this series of articles I’ll present the findings from scientific research published in peer reviewed journals which details the value of business simulation across multiple domains.

A study conducted recently by Ranchod et al. (2014) built on previous simulation research by testing the use of a high-quality computer-based business simulation. In particular, their focus was on three main categories of learning outcomes built on Bloom’s taxonomy with associated learning goals defined as:

i. Cognitive (understanding and retention at conceptual, procedural and strategic level):

• teach students the terminology, concepts and principles of business in general or of a specific discipline;
• help students understand the interdependence between various business functions (marketing, finance, production, sales
• demonstrate the procedural difficulty of applying business concepts in complex realistic situations; knowledge retention.

ii. Behavioural (skill practice and development):

• enable students to implement course concepts, by taking decisions and experiencing the consequences of their actions in an interactive environment;
• improve students’ team work and relational skills;
• generate practical experience in taking and implementing business decisions;
• improve students’ analysis and decision skills.

iii. Affective:

• improve student attitudes towards the discipline;
• enhance students’ motivation and engagement;
• increase students’ satisfaction regarding the learning experience.

In essence this relates to simulation games as experiential learning (discussed in this blog here) where a Lewinian learning cycle is followed. This means:

0. “the interaction with ‘concrete experience’ leads, through ‘reflection and observation’, to the ‘formation of abstract concepts and generalisations’”

In their study of the literature the researchers noted that there was significant evidence from earlier research in each of the three domains and they stated:

• Students can develop a deeper understanding of fundamental business concepts and procedures, as well as of their strategic significance, during their interaction with the generated experiential situation.

• Previous studies report a positive impact of experiential learning methodologies on students’ skill acquisition. Proposing an engaging, dynamic and interactive learning environment, business simulation games put students in a situation of ‘learning-by-doing’, while the realistic representation of the simulated business systems ensure the transferability of the acquired skills in real-life situation

• Besides cognitive and skill-related outcomes, simulation games research indicates the existence of affective outcomes, expressed through increased motivation, positive attitudes towards the simulation game experience, engagement, general satisfaction, and enjoyment

A critical aspect of the research indicated the dynamic nature of learning outcomes where a causal relationship exists between them. This means individuals are able to formulate their own concepts and generalisations having tested the implications of their knowledge in new situations within the simulations – the basis for skills development.

In fact, the article cites evidence of ‘adaptive expertise’ which is distinct from ‘routine experts’ who solve familiar problems quickly and accurately in that adaptive experts can “innovate and create new procedures to adapted to novel problems and situations”.

The next article in this series will look at the first of the learning goal domains: Cognitive and Conceptual Understanding.

Source:  A. Ranchhod et al. / Information Sciences 264 (2014) 75–90

 

External Validity and Sense-making

Despite the growth in business simulation use they remain a relatively sparsely utilised resource. In my experience, an encounter with them leads participants fairly rapidly from positions of scepticism to one of positive engagement. Not only do they enjoy them, but they typically feel tested and so enlightened in some regard. 

 

Yet, as with all learning experiences there is a desire to focus on what the scholars and researchers call ‘external validity’  which, in the case of simulations, typically translates as “have they learned anything they can actually use in their day-to-day work lives?”. This question is an important one, of course. But it is inherently devilish to answer and so leads to skittishness on the parts of those who have to justify its appearance. In many ways this issue is present in just about every learning instance you can think of yet the same rules do not always apply.

 

How many lengthy presentations or densely packed reports does the typical executive encounter in a working week? How much of that information is retained and, more critically, how much of it is actionable? Precious little we can presume. As humans we are selective when it comes to processing information and we have an in-built tendency to prioritise the data which means something to us or which we already value. This is unique to us all. My values will differ to yours, as they will in turn to those you work with. In such an environment how can any of us be sure what we ‘learn’, in whatever circumstance, is of value? 

 

Business Schools and MBA programmes place a premium on mixed method deliveries where case studies exemplify theory and theory is explained in context. The abstract made tangible and the tangible explained. But how can we be sure that any of this leads to improved decision making? To strategies which are informed by past events and which offer rational, deliberate actions responsible for value creation in organisations? The fact is we don’t. At least not in the sense that scholars would call valid. In many ways this accounts for what Mintzberg called ‘emergent’ strategy in his seminal work, where strategy becomes something dynamic and malleable. An unending process where what was intended is rarely achieved and where results validate a successful managerial intervention regardless of how dissimilar it is to its original form. Failure, conversely, is typically the result of external environments.  

 

The brilliant Karl Weick has built a prodigious body of work which exposes our natural tendency for what he terms ‘sensemaking’: our retrospective accounts for what has occurred which gives meaning to experience. One of its most intriguing elements is the observation that we favour plausibility over accuracy. We are literally guilty of explaining events away because our version seems more likely. This is a form of cognitive bias that has been proven many times by psychologists. In the context of change and its design Weick describes it beautifully:

 

“There was not a transition from imagination, through intention, into execution. Rather, there was an imaginative interpretation of execution that imputed sufficient coherence to the execution that it could easily be mistaken for an intention.”* 

 

His point is that the change may occur before the decision to instigate change became explicit. But our desire to make sense of our current environment leads us to view the world through the lens of rational intent. A dangerous assumption. When faced with scarce resources and multiple alternative learning opportunities HR professionals are driven to rationalise decisions by applying the same tools that exist elsewhere in the organisation such as ROI or other quantitative measures. Yet, as we know intuitively, training an employee to use an IT system or to acquire a skillset is fundamentally different to leadership or business acumen development .

 

It is possible to disarm this argument over external validity by recognising first that we do not learn in a vacuum and what is available to you now, in this minute, may be the most useful of all things you need to remember and use. At what point you acquired the knowledge you used to make sense of it, the medium it arrived in, and the circumstance which situated the learning is largely unimportant. You make sense of business scenarios which require a decision by marshalling the sum of your experience and actions to date. At all times. 

 

In that context, the point must surely be: keep learning. Keep exposing your mind and body to the broadest possible range of scenarios for you  cannot know what is around the next curve. If you want your high-potentials to grasp some of the issues and decisions which a senior leader makes without handing over the reins to the company there has to be an alternative.

 

Simulations are contextual. They situate you in a place and time and ask you to make sense of it. They provide you with data and are usually charged with emotive potential. All of the opportunities to learn are contained within them if you are open to the experience. And, perhaps, there will come a time when a decision is reached and the will that you drew on came from somewhere you could not place. No doubt you’ll have an imaginative interpretation to hand…

 

*(Weick, 2001, p. 62)