Teaching Complexity : The Limits of Evidence and the ‘ Prospective ’ Case Study

Policy practitioner–students in a Master of Public Policy programme in New Zealand describe many problems and processes in their work environments as ‘complex’. Yet, they hold firmly to a belief in the merits of ‘evidence’ to guide their advice and decision making in the face of that complexity. This article examines the aims and pedagogy of a two-course sequence designed to help students replace over-reliance on analysing existing evidence with understanding of the ways complexity concepts can aid in estimating possible outcomes of policy interventions. Starting with identifying evidence challenges, students learn how to compare a status quo situation and a prospective case of that status quo in which a new policy has been implemented. This method draws on existing scholarship in lesson-drawing for policy applications. Students are eased into an appreciation of a variety of complexity frameworks and concepts by looking at a case about which there is, strictly speaking, no evidence.


Introduction
Students in Victoria University of Wellington's School of Government Master's programme in public policy and public management are primarily working professionals who return to university for study.The programme brings students of both policy and management together for a greater part of their learning, recognising the interdependence and integration of the two fields in practice.Students in both degrees share three core courses (out of 12 required for the degree) that cover the overall context, institutions and dynamics of government and governing, national and international societal and economic forces that influence the essential features and interdependence of management and policy, and theories and practices in policy and management.
The common core courses also recognise that practitioners need to be able to critically reflect on their specific practices and to have high-level intellectual capability to be both critical and creative in their work.As designed, the programme ensures that classrooms have a great diversity of learners.The courses are interdisciplinary and problem-focused, drawing on selected examples and-importantly-ongoing challenges in students' workplaces.The explicit intentions are to increase students' understanding of the relevance of many disciplines to their work, and to sensitise them to the challenges and expectations on them as professionals in this environment.Much of the learning in these courses is relevant for advancing students' understanding of the complexities in policy development and implementation, but 'complexity' is not directly addressed as a theme.Nevertheless, from the start, students learn in an environment in which they are assisted to look at their practice, in primarily policy or management, openly and reflectively.
In addition to the common core courses, students pursue a four-course core (comprising one-third of the degree) specific to management or policy, and complete their degree with shared electives and independent research.Students in the Master of Public Policy stream study a twocourse sequence, which I teach with contributions from colleagues.Teaching complexity in these two courses is the focus of this article, with a particular emphasis on using the device of a 'prospective case'.
In the next section, the policy cohort and their initial orientation to study are more fully described.Following this, my approach to teaching complexity is described, with reference to the course learning objectives, content and pedagogy, including attention to the rationale for using a prospective-case method.A final section presents some reflections on the methods used and their merits in teaching complexity to a cohort of experienced policy professionals.

Master of Public Policy Students' Initial Orientations to 'Evidence' and 'Complexity'
In their workplaces, policy students are exposed to strong rhetoric and prescriptions concerning the supremacy of 'evidence' for developing and implementing policy.Consequently, they are intensely distrustful of any information that looks 'subjective' or 'biased'.They applaud claims such as one in a report from the US National Research Council: "[W]hen the question on the table is what are the 'real' conditions or what will 'probably' happen if we implement one policy instead of another, science is on balance a more dependable and defensible guide than informed hunches, analogies, or personal experience" (Prewitt, Schwandt, & Straf, 2012, p. 10).Closer, to home, they may be employed by a departmental chief executive, who proclaims: "We have to make sure what looks like a good policy idea is backed up by solid evidence and quality analysis".In workplace discourse, it is equally broadly claimed that policy formation and implementation is 'complex'.Usually this refers to the presence of uncertainty and value clashes, with many inputs and actors at the interface between policy analysis and politics.And while there may be talk of 'interdependencies' or 'unpredictability'-terms that are used in some complexity literature-the typical response is to seek more or better evidence, as the then head of Australia's Productivity Commission urged: Without evidence, policy makers must fall back on intuition, ideology, or conventional wisdom-or, at best, theory alone... .But the resulting policies can go seriously astray, given the complexities and interdependencies in our society and economy, and the unpredictability of people's reactions to change... .policies that haven't been informed by good evidence and analysis fall more easily prey to the 'Law of Unintended Consequences' which can lead to costly mistakes.(Banks, 2009, pp. 4-5) Thus, the way many students appreciate their work environment creates a significant block to introducing a deeper understanding of 'complexity'.Accordingly, I adopt a wellestablished pedagogical practice, which holds that for students to learn a new concept or skill, the teacher must first activate some doubt or discomfort with the students' pre-existing beliefs, then present new material, provide examples, and offer students an opportunity to practice applying the material.Over the two courses, I aim to help students enlarge their skills and to understand that (a) there are significant limits to using evidence (in part since it is by definition about the past); (b) they already compensate for these limits using various natural, judgemental (and hence, subjective and biased!) processes that help them to see a little bit into the future; but (c) they don't have to give up entirely on the rigour and reliability promised by evidence; since (d) skills for working with complexity can complement what they gain from conventional evidence analysis.

Learning Objectives and a Stepped Introduction to Complexity
Most students undertake study in economics and an introduction to policy analysis before enrolling in the remaining two courses in the policy core.The course prescriptions for the two subsequent courses refer directly to complexity: Policy Methods and Practice: Qualitative and quantitative techniques for collecting, analysing, interpreting and applying information and evidence to advance policy objectives particularly under conditions of complexity and uncertainty and in light of given task requirements.
Policy Workshop: Integrating theories and practices to produce in-depth studies of selected complex policy issues with a focus on the design and evaluation of alternative policy options.
In the Policy Methods course, the specific learning objectives are framed in a way that accepts that learners (and often their employers) are interested in gaining 'conventional' evidence skills, such as determining the specific evidence requirements for a given policy research or analysis task; selecting or adapting methods and practices for various challenges; demonstrating critical awareness of the strengths and limitations entailed in applications; and judging the adequacy of evidence.Consideration of complexity, however, begins in the first session.
The first exercise is a round-robin to elicit qualities of a 'good policy professional', designed to activate some scepticism about the reach of 'evidence' in practice.As expected, everyone knows some analytical whiz, and first-round propositions are heavily weighted toward analytic skills, such as the ability to define the problem, to find and assess information or to conduct cost-benefit analyses.Soon come ideas about relationships in the environment, such as dealing with political constraints, attending to international norms, and communicating with different demographic groups or other professionals.Eventually, a few students venture some qualities consistent with managing complexity and uncertainty.For example, they mention making sense of a messy situation, or intuiting the inclinations and likely responses of a politician and other stakeholders.Often, a student will also volunteer characteristics that seem more directly to acknowledge complexity, such as the ability to anticipate or be alert to 'unintended consequences' or to the unpredictability of people's behaviour.However, in offering these suggestions, students may be looking through the prism of an 'evidence' or analytic frame, consistent with the Banks quotation above.By the end of the round-robin, students are more attuned to some tension in their use of 'evidence' and ready to consider my initial definition of complexity: as a condition in which the knowledge required to bring about a desired change is not readily mustered by even the most sophisticated analysis of evidence, but is rather highly distributed in the communities, organizations and individuals where change is to take place and is unknowable at the point when policy decisions are being made.
Although students arrive wanting to become better evidence-handlers, the exercise forces a small wedge into the door to complexity learning.Students realise for themselves that there is more to their practice than developing evidence products, such as analyses and 'objective' advice.With the scene thus established, student examine more carefully the assumption of the 'science' framing of evidence, drawing on arguments for the variety of information that aids decision makers (Head, 2008), the connection between complexity and the tradition of American pragmatism (Sanderson, 2009) and the distinction between information and evidence, as described by Majone (1989, p, 7): Evidence is "information selected from the available stock and introduced at a specific point in the argument in order to persuade a particular audience of the truth or falsity of a statement".Students consider that for many policy challenges, there are unfillable gaps in the evidence base.In their first written assignment, students map out some of the 'evidence challenges' in a relatively discrete topic of their choosing, taking into account value differences, and various contextual features.
Hard on the heels of these efforts to call evidence into question, the class considers definitions of uncertainty, ambiguity and complexity and is introduced to methods and practices to address them.I present three 'complexity frameworks' (see Table 1), each of which takes a different tack: Eppel's lens (2012), developed from the complexity literature, Pawson's (2012) suggestions for 'mapping' a pending policy intervention, and Room's (2011) advice to an 'agile' policy professional.I illustrate aspects of these frameworks using a published case study, such as Smith and Lawrence (2014) on food security and resilience in Rockhampton, Australia, following flooding that cut off a community's access to food.Such case studies are ubiquitous in the published literature.True to form, such articles have some theoretical point to advance, and it is this that makes them generally seem 'irrelevant' or 'overly academic' to our cohort of students.Yet, almost all such articles contain detailed descriptions of a case, which can be read on their own, and are generally suited to get the reader's complexity thinking going, with complexity frameworks providing direction and structure: What systems interacted?How did prior changes in the regulatory system change the local response to the drought?What information could have been known about the risks to the supply chain but was available only in retrospect?Finally, the students read Mary Schmidt's (1993) story of the failure of the Teton Dam to further sensitise them to the alternative forms of knowledge potentially available to for decision-making, but which are generally dismissed from the 'evidence' base.
Students are allocated one 'theme' from Table 1 or from Schmidt's types of knowledge, which they consider with respect to the policy topic they are investigating.Class presentations ensure exposure to varied applications.The rest of the Policy Methods course covers evidencehandling skills through units on various methods of collecting, analysing and interpreting evidence and assessing its qualities in specific applications.This treatment reinforces the merits of good analysis, which is not obviated by complexity.

Learning Complexity through Prospective Case Analysis
In the Policy Workshop, students learn to draw and apply transferable lessons from an exemplar (Wolf & Baehler, 2017).They then undertake a realistic, if somewhat attenuated project that demonstrates their ability to apply these methods, as well as other learning from throughout the policy core.Building on the Policy Methods course (which activates doubt about the full rhetorical impact of the 'evidence-based policy' mantra, awakens recognition of some natural aptitude for complexity thinking, and offers exposure to numerous examples of how aspects of complexity may influence policy analysis and decision making), I continue in the Workshop to gradually draw students along a learning pathway in a structured manner.
The high-level tasks in the Policy Workshop start with a status quo 'situation analysis' of a current policy challenge, which can be 'cased' as a policy status quo, or receiving environment (or 'target case' in the policy transfer and learning literature) or simply A, with attention to the stakeholders, problem-and solution-framings, key contextual influences and tensions among them.Next, students make a detailed study of a promising policy intervention that is absent in A, but in place elsewhere (B).By designing an intervention adapted from B for application in the receiving environment (A), students create a prospective case (A′), which is A at time t + 1, in which policy based on insights spawned by B is producing impacts, for better or worse.Such prospective-case thinking aids students in taking a broader perspective-to think about the policy ecosystem, comprising actors and resources and, importantly, other policies, and to imagine the introduction of new decisions.In sum, to estimate the impacts in A′, students take into accountat some level and selectively-aspects of the complexity frameworks in Table 1.Room (2011) In the classroom, time is taken to motivate the reasons as I see them for taking a comparative-case approach and to justify the place of these methods in a complexity-ready toolkit.As writers have made clear (for example, Thomas 2010, 2011, rehearsing the arguments of MacIntyre), for one reason or another, generalisation does not work easily in the social sciences, which is a problem for the application of evidence to new decisions.No two particular instances of a policy are ever similar enough, and the differences always cast doubt on the applicability of the generalisation.While replicating studies in many different locations under a diversity of circumstances can help reduce problems of external validity, the high costs and long timeframes associated with repeating sophisticated studies make this prohibitive most of the time.Even with replication, problems of external validity tend to persist because social problems, implemented solutions, and contextual settings vary in so many different ways from case to case.
In the Workshop, following their situation analyses, students find an exemplar elsewhere, assess it to gain an understanding of that other case, then abstract and adapt learning for application in the presenting case.Students read Rose (2002) and Barzelay (2007), which offer methods showing how practitioners and researchers can use existing policies or management processes as cases for learning, based on the congruence between studied cases and prospective decisions.Importantly, these methods are intended for application within a causal-analytic frame, and so they do not entail students making a cognitive alteration for which they are philosophically disinclined.'Lesson-drawing' and 'vicarious learning' methods allow for the transportability and adaptation of "fresh thoughts" (Rose, 2005), based on a causal model of the 'known' and the ability of policy learners to "use their minds" (Barzelay, 2007) to make an effective transfer.Nevertheless, some discussion in the classroom gently critiques these two sources, providing students with a critical awareness of the challenges complexity offers in the search for 'fresh ideas' and the judgemental elements of 'using one's mind' in the course of deriving and transferring lessons (Wolf & Baehler, 2017).These include the manner in which relevant fresh thoughts are generated and detected; the enormous number of variables that differentiate cases in unknown ways; and the fact that lessons are from the past but for the future, all of which complicate using one's mind in any causal-lesson-drawing activity.
While students are encouraged to undertake some analysis (for example to estimate costs of implementing the initiative in the new context based on the experience elsewhere), they have little time for actual analysis, and thus they are (intentionally) thrown into the need to make assumptions and judgemental leaps, to make use of analogical thinking, to fall back on guessing.Additional learning exemplars are discussed in class, and guests also present some 'real' cases involving policy transfer.For example, the Wellington City Council Community Services Manager has been invited to explain the city's homelessness strategy.Wellington follows a 'housing first' strategy, based on some overseas experiences.This compares with several Australian cities that have followed a variant of this strategy, known as 'Common Ground' (but for which there is scant 'evidence'; see Parsell, Fitzpatrick, & Busch-Geertsema, 2014).I look for published case studies on policy issues that are socially embedded (rather than concerning a technical matter).The implementation of Stockholm's congestion charge (a good overview is Börjesson, Eliasson, Hugosson, & Brundell-Freij, 2013), for example, can be used to think about a prospective case in Auckland, which is also searching for ways to address congestion and faces the sort of resistance documented in Stockholm.
Another example (Steyaert et al., 2007) is used to demonstrate the manner of 'judgemental leaping' from the presented case, which is very specific, to a very different prospective case.The authors narrate an example of community-driven ecological restoration of marshlands along the French Atlantic coast.Maraîchine cows, a local breed, had lost favour during agricultural intensification because of their low milk-and beef-production levels.A farmer fond of local history and marsh heritage began to look for the last Maraîchine specimens and to work towards rebuilding the population by identifying individual cows, finding farms to house them, organising breeding and so on.In the process, the farmer brought together agricultural schools and ecological museums, but also some regular landowners and farmers.This network of people shared the objective of preserving the marshland's biological resources.The lesson from this example relates to preserving species and diversity while also providing for community resilience: the case details (French cows on the Atlantic coast) matter less than the somewhat more abstract storyline.Making a connection requires someone (roleplayed by students) with an actual community-resilience challenge in mind to generate incipient hypotheses ("X here functions like the farmer there").As further elaborated below, readers act as if they are engaging in a hypothetical dialogue with a scene that reading helps them to visualise and which in turn allows for the emergence of "pursuit-worthy" ideas (McKaughan, 2008) for application in the home environment.Reading such cases without the theory parts (like reading a regression study while skipping the numbers) can inspire fresh ideas (uninfluenced by the author's theoretical objectives).
Time is taken to consider what makes a 'case' and how a case may be complex.I use an ecosystem metaphor to draw attention to interactive systems and feedback loops, emergence, adaptation and co-evolution.I also reflect that cases tend to be relayed in narrative form, which helps to overcome static and single-perspective framings.I emphasise that in policy, complexity is typically not found in statements of 'problems' or 'solutions', most of which appear entirely straightforward-such as "children consume too much sugar so we need tax sugar-sweetened beverages".Rather, it is the interactions in the world-among varied understandings of the issues, interpretations of the evidence and values-that make successful policy implementation so challenging, and that moreover, these interactions are not just hard to analyse, they are complex and cannot be known or estimated or modelled well at all.
Instead of trying to overcome these obstacles with the brute force of analytic thinking, students are invited to try to relax that tendency in order to approach cases simply as storiestales of events and actors and outcomes in unique contexts, since complexity renders such cases unknowable analytically.Yet, a policy or management practitioner, with some relevant experience, can observe and gain insights using a combination of abduction and phronesis.
Briefly, abduction is a form of inference originally proposed by Aristotle, but significantly developed by Charles Peirce.It happens in a case context, when an anomaly is noticed in a stream of observations.Noticing something that stands out leads to one or more plausible explanations (Douven, 2017).Cases can raise a great number of abductive explanations, so there is a need to gauge their usefulness.Analogical reasoning can help.In analogical reasoning, a comparison is made between two domains that highlights similarities and supports a hypothesis pertaining to something as yet unobserved in the second domain, but observed in the first (Bartha 2013).Finally, phronetic reasoning can supply the grounds for applying knowledge to a new problem, especially when the ability to draw analytical inferences through 'objective' scientific reasoning is curtailed (Thomas, 2011).
Again, we can look back to Aristotle, who introduced phronetic reasoning, a process that starts in the middle of already knowing, with the knower's biases (prejudices) essentially opening up to what is next to be understood.Thomas (2011, p. 24), drawing on others, describes phronesis as involving a "combination of knowledge, judgement and taste, together producing a discernment, through which we see links, discover patterns, make generalisations, create explanatory propositions . . .all the time emerging out of our experience".As elaborated by Gadamer (see Thomas, 2011, p. 21), another case creates in a person an 'image' that is tentative, but sufficient to be seen and heard in the context of another's experience . . .but then used in the context of one's own: it is essentially visual and dialogic.
Although travelling to the site of the case of interest is desirable, it can be impractical.Students, however, drawing on published documents, evaluations and academic case studies can extract a great deal of the basic story, be it about congestion-relief in Stockholm or speciespreservation in France.Following Gadamer's observation, the student-practitioner can see and hear another story from their own vantage point.That is, the exemplar enables a person to 'see' the prospective case by a process of iterating between what can be assessed directly and that which can only be anticipated or seen in the 'mind's eye'.As noted by Schön (1983), professionals have a capacity to undertake a "conversation" with a past exemplar, to carry a particular result of that conversation forward as a lesson, and then to apply reflections prospectively to the next assignment.If a learner approaches unique cases, suitably prepared by an initial 'situation analysis' and with a view to triggering plausible hypotheses through the process of abduction, resulting hypotheses can then be reflected on in light of their implications for action (via phronesis), rather than their fit with any posited pattern, such as a theory or diagnostic category (Thomas 2010).There is, of course, no certainty that the 'right' lessons will be learned, but this situation affects all learning.

Learning Outcomes and Further Opportunities
Many of the methods used in prospective case analysis can continue to be used after a change is implemented, by iterating a revised prospective case accounting for new observations and hypotheses.After the course, it is hoped that students will be inspired to undertake complexityaware behaviours, such as monitoring for anomalies (Room, 2011) in newly changed policy ecosystems, and continuing to mentally form and test hypotheses (Sanderson, 2009;Eppel, Turner, & Wolf, 2011).With such an 'experiment-minded' perspective, grounded philosophically in pragmatism, a practitioner searches for what does not fit the expected pattern, followed by actions (including 'thought experiments') that take into account new perspectives arising from plausible explanations about what is occurring.
Some indicators that students have learning about complexity emerge in discussion and through signs that there is greater sensitivity to complexity when approaching challenging policy issues, and perhaps somewhat less insistence on 'science'-oriented information when making decisions looking ahead.After the final presentations in the Policy Workshop, students individually record tips for others on post-it notes.Ideas are volunteered that recognise complexity, ambiguity and uncertainty, as well as the unavoidable contextuality of policymaking.Sample tips include: • Dealing with ambiguity: sometimes a 'best guess' is the best you can do-use own insights and judgements to inform as well as any literature • Bear in mind: context, context, context • Learn from what is out there • Different types of knowledge (e.g., tacit, intuitive) are valid • Use evidence and judgement: sometimes all you will have is your 'best guess' In sum, I hope, but cannot be sure, that students complete these case exercises with their faith in evidence still intact, but also prepared to reject sharp dichotomies, to value more highly varieties of knowledge, and to welcome the emergence and evolution of their own understanding within the constraints presented by their practice environments.Initial support for this desired outcome is shown in an assessment, mentioned earlier, at the end of the Policy Methods course.Students are allocated one 'theme' from Table 1 or from Schmidt's (1993) types of knowledge.