Complex systems

Systems are combinations of elements that interact. A distinction is often made between “complex” systems and systems that are “simple” or “complicated” [6–8]. What make complex systems unique are a number of attributes, including nonlinearity, their dynamic and unpredictable nature, and the ways in which they co-evolve with their environment and produce emergent outcomes [9–11]. Elements within a complex system (for example, individuals, organizations, activities, and environmental characteristics) interact with each other and are connected in nonlinear ways [6,12–14]. Over time, the behavior of system elements leads the individual elements and the system as a whole to adapt and co-evolve with the broader environment—that is, the system is dynamic [6,7, 12,13]. There may or may not be a central authority within the system, such as a president, local authority, or management team, but a complex system is assumed to adapt and behave in ways that cannot be reduced to simple, organizational hierarchies. Because of this, a complex system and its elements are considered to be self-organizing [6]. The individual interactions among system elements collectively generate emergent, system-level behavior wherein the system displays attributes that cannot be reduced to its individual parts [2,6,12,15].

Research into complex systems takes place across academic disciplines and has roots in both systems thinking and complexity science. Although often grouped together because of some conceptual similarities, systems thinking and complexity science can be considered as distinct yet overlapping traditions [16,17]. Systems thinking may be best described as an orientation that prompts researchers to take a holistic, rather than reductionist view, of phenomena and study them in the context of their real-world systems that are open to and interact with surrounding systems. Systems thinking draws on theories, concepts, and methods from a range of disciplinary fields [18]. Complexity science, on the other hand, is more strongly rooted in the mathematical sciences and has drawn on complexity theory, which emphasizes uncertainty and nonlinearity, to create and refine specific methodological approaches to modeling complex systems in order to estimate and predict their emergent behavior over time. Systems thinking prompts researchers and practitioners to consider the boundaries of the system they are studying or in which they are working [19] and places an emphasis on the interactions and relationships between system elements and the system with its broader environment [1,6]. Further applying concepts from complexity science prompts a consideration of how those interactions create nonlinear chains of cause and effect, are unpredictable, unfold overtime, and give rise to system-level emergent outcomes [20].

Complexity has been part of the vocabulary of public health evaluators for decades [16,21]. However, public health evaluations have tended to focus on the complexity of interventions rather than of the systems within which interventions are implemented [22]. A “complex intervention” is one that has a number of interacting parts, targets different organizational levels or groups of people, and aims to affect a number of outcomes [16,17]. In contrast, a complex systems perspective considers complexity as an attribute of the system. The intervention itself may also be complex, for example, a coordinated program of interventions that affect different parts of a system. However, simple interventions can also be theorized to have complex consequences if they are implemented within and interact with a complex system. For example, a single change in a law affecting the price of products that affect health (such as an alcohol or sugar sweetened beverage tax) can be described as an (initially) simple intervention that quickly becomes connected to a complex chain of interactions between industry, retailers, public opinion, consumer behavior, media and policy—each of which may have an impact on future implementation and effects of the intervention itself [15,23]. The way a complex system responds to an intervention may lead to emergent consequences that could amplify or dampen the intervention’s impacts, change the characteristics and behavior of the system over time, and affect future decision-making [15,24]. From a complex systems perspective, the role of the evaluator is to make sense of the interplay between the complex system and the (simple or complex) intervention to help explain health and other impacts and inform future decisions about implementation [1].

Process evaluations and qualitative methods

Traditional evaluations of simple or complex public health interventions often focus on measuring impacts on a single (or small number) of prespecified health and health-related outcomes [10]. However, impact evaluations alone offer little opportunity to explore the mechanisms behind an intervention’s success or failure, particularly when impacts are unevenly distributed among different population groups. For this reason, other forms of evaluation, particularly process evaluation, have been developed and utilized in order to understand intervention implementation and the mechanisms by which interventions may lead to impacts across a population [17,25]. There is no single definition of a process evaluation, but the Medical Research Council’s (MRC) Guidance on Process Evaluations of Complex Interventions argues they “can be used to assess fidelity and quality of implementation, clarify causal mechanisms, and identify contextual factors associated with variation in outcomes” [26 p. 30]. A process evaluation is often, although not always, conducted alongside an outcome or impact evaluation that quantifies the impact of an intervention on a range of outcomes [16].

Process evaluations of public health interventions may benefit from an explicit adoption of a complex systems perspective. The application of systems thinking and insights from the complexity sciences can provide a means through which to evaluate and understand the nonlinear ways in which interventions may lead to a number of impacts within a system. This could include impacts considered to be of interest when the evaluation is initially planned and impacts that emerge as potentially important as the evaluation progresses. By bringing an explicitly relational focus to the evaluation design and placing the wider context in the foreground of the analysis [24], a complex system approach to a process evaluation may help to make sense of intervention mechanisms within a real-world context. An explicit complex systems perspective may also help evaluators construct a narrative that explores the trajectory of a given system. This could include considering how the intervention acts as an event that prompts a series of changes in the way a complex system behaves [15]. Furthermore, it could include consideration of how the intervention itself changes, as system elements and the system as a whole adapt and respond to it [15,24].

Although process evaluations can include quantitative assessments of intervention outputs, they typically draw on a range of qualitative methods. Qualitative methods are well suited for unpacking complex causal chains, understanding changes in implementation, representing varying experiences of the intervention, and generating new theories to inform future decision-making [17]. Proponents of explicitly using complexity theory within qualitative designs argue doing so “has potential to capture and understand complex dynamics that might otherwise be unexplored” [27 p. 3]. Bringing a complex systems perspective to a qualitative process evaluation could have a range of methodological implications. For example, it could involve mapping the system of interest, a sampling strategy that seeks to recruit participants relevant to different parts of that system, a form of data collection geared towards assessing relationships within a system, and an analysis framework that incorporates concepts drawn from systems thinking and complexity science.

There is a large body of literature on quantitative methods for complex systems approaches and some examples of such methods being applied to the study of policies and interventions that may affect population health [28–33]. Many of these approaches build simulation models that estimate and predict the impact of interventions on outcomes of interest [34]. These approaches have been developed within the complexity sciences and include methods such as system dynamics modeling, microsimulation modeling, and agent-based modeling [3,20,35,36]. Although these methods may begin with some qualitative work, such as participatory workshops to map a system of interest, their aim is to generate quantitative estimates of future or hypothetical impacts [31]. Compared with quantitative methods, there is little consensus, and less has been written on how to explicitly draw on a complex systems approach for process evaluations that use qualitative methods. This represents an underdeveloped area for complex systems evaluation.

This systematic review therefore aimed to identify the concepts and methods currently used in public health evaluations that apply a complex systems perspective to process evaluations involving qualitative methods. Specifically, this review sought to answer 3 research questions: (1) What types of public health interventions have been subjected to process evaluations that use qualitative methods and apply a complex systems perspective? (2) What are the qualitative methods used in this body of literature? (3) What concepts and theories associated with complex systems are used in process evaluations that use qualitative methods? Drawing on this body of literature, we then had a secondary aim of developing a framework for qualitative process evaluation from a complex systems perspective. We sought to develop an evaluative framework that researchers (working in academic or practice settings) can use as an overarching structure to guide evaluative efforts [37]. In our Discussion section we therefore present our framework and provide some guidance for researchers on the potential role of qualitative data in identifying and understanding aspects of complexity within process evaluations.

Data sources and screening

Relevant process evaluations were identified through several different search methods. First, we conducted an expert consultation whereby we contacted 32 academics with an interest or experience in complex systems thinking and its application to public health and asked them to identify any relevant examples of complex systems evaluations. The academics were identified through an ongoing familiarization with the literature on complex systems and public health, as well as through our own professional networks. In the original consultation, we did not request permission to be named, but those who did provide permission during the review process are named in the Acknowledgments. We then identified 2 relevant systematic reviews on systems thinking and public health [35] and complexity theory applied to evaluation [20]. From the studies identified in these reviews, we selected evaluations that met our inclusion criteria (next). Finally, we conducted an electronic search covering January 1, 2014–September 30, 2019 using 3 databases: Scopus, Medline, and Web of Science. The search dates were set to capture evaluations published after the 2 systematic reviews. The electronic search strategy included terms and synonyms for systems thinking, complexity science, evaluation, and public health and was restricted to English-language publications. An example of the full search strategy can be found in S1 Text. This study is reported as per the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guideline (S1 PRISMA Checklist).

Titles and abstracts were screened initially by one reviewer, and all potentially relevant studies were independently screened by 2 reviewers. In cases in which a decision was not clear cut, or the reviewers disagreed, a discussion was held with a third reviewer. The review had 4 inclusion criteria, which we describe in more detail next. In brief, studies were included in the review if they (1) self-identified as taking a systems- or complexity-informed approach; (2) were relevant to public health; (3) were process evaluations of interventions with empirical findings; and (4) utilized qualitative methods.

Studies were eligible for inclusion if they self-identified as using a systems and/or complexity perspective at any stage of the evaluative process, including during the design, data collection, analysis, or interpretation phases. We took a broad view of public health to include upstream determinants of population health, which include alcohol, the built environment, community health, community safety, education, employment, environmental health, food, health promotion, housing, illicit substances, obesity, policing, regeneration, sexual health, social welfare, tobacco, trading standards, transport, and urban planning. Studies that covered topics not included in the aforementioned list were considered if they concerned population health; decisions in these instances were made between 3 reviewers. Studies concerning treatment in health service settings were excluded. Studies were only included if they reported empirical findings of a process evaluation; protocols and discussion pieces describing evaluations without presenting results were excluded. Process evaluations alongside outcome evaluations were eligible for inclusion, although our analysis focused solely on the process evaluation component. Finally, studies were eligible for inclusion if they used qualitative methods, which included interviews, group interviews or focus group discussions, (participant) observation, document review, free form responses on questionnaires, and participatory and visual methods, including for example, mapping workshops and photography. Evaluations employing mixed methods (wherein qualitative data were integrated into the assessment of the intervention alongside other methods) were included, as long as there was a substantive component that generated and analyzed qualitative data. To operationalize this criterion, we considered the ways in which the mixed methods research was designed, and we included studies that generated qualitative and quantitative data concurrently to evaluate an intervention (triangulation design); studies in which the researchers primarily utilized a qualitative design with some supporting quantitative output or outcome data (embedded design); studies in which the qualitative data were used to make sense of intervention outcomes (explanatory design); or studies in which qualitative research was used to generate hypotheses about the intervention that could be tested quantitatively (exploratory design) [26,38]. Studies utilizing these mixed method designs were eligible for inclusion even if the authors did not label the design or describe the rationale for the chosen approach. A substantive qualitative component referred to the authors both describing the qualitative methods, including data collection and analysis, as well as presenting qualitative data. Covidence software was used to help facilitate the screening process [39].

Data extraction and synthesis

The analysis began with an in-depth reading of, and familiarization with, the included studies, with specific attention paid to the ways in which they drew on systems thinking and/or complexity science and the methods utilized to achieve their evaluative aims. Data were extracted on each study using a template designed for this review. Specifically, data on the study’s research question, public health area, country, intervention, the application of complex systems thinking, the methods and analytical approach, and system map (if presented) were extracted (see Table 1). The “complex systems perspective and evaluation stage” column shows how systems thinking and/or complexity science featured in each evaluation and at which stage in the evaluation (i.e., design, data collection, analysis). The system map column reports the studies that included a map of the system and describes what the map detailed. If the evaluators published a logic model, it is noted in this column. Where studies gave rise to more than one publication, we considered them “linked” and extracted data from across the identified studies. The data extraction process was completed by one reviewer and double checked by a second.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of the included studies.

https://doi.org/10.1371/journal.pmed.1003368.t001

Alongside the data extraction process, a list of concepts from systems thinking and complexity science was generated through an ongoing familiarization with these bodies of literature. A number of papers and books that are frequently referenced within the public health literature on complex systems were selected during this familiarization period [1,6,7,9,12,15,22,40], and from this, a master list of systems and complexity terms was generated. Our aim was that this list captured the key principles associated with each of the traditions and could be used by those wishing to gain a familiarization with systems thinking and complexity science. We found that not all authors describe the same concepts within these traditions and they often use different language. As a result, there was a subjective element to generating the list with the research team making choices about which concepts to feature and how to define them. In particular, although many authors describe “context” as a key systems thinking concept, and we initially also included it in our list, we ultimately chose to exclude it due to its substantial overlap with many other concepts. “Context” describes the factors in the environment that affect the system, particularly historical, temporal, geographical, political, and social factors [13]. As a result, arguably the entire system represents the “context,” and it therefore does not represent a meaningful category when trying to describe and analyze a changing system. In addition, we recognize that there is conceptual overlap between many of the concepts and that the boundaries between them may be somewhat fluid. In the Discussion section a glossary of terms and how they might be applied within a process evaluation using qualitative methods are presented.

Critical appraisal

No tools exist to assess the quality of process evaluations informed by a complex systems perspective. Therefore, for this review, we critically appraised how systems thinking and complexity science were employed in each paper. Specifically, we assessed the degree to which each study identified through the search strategy described, captured, measured, or applied each concept in a meaningful way. The decisions were depicted using a traffic light color scheme. A green color code was applied when a study explicitly applied a concept at any stage of the evaluation process, including the design and planning stage, data collection, analysis, or interpretation. For example, a study would receive a green code if it explicitly described the boundaries of the system under inquiry at any stage in the evaluation. Evaluators might use the idea of boundaries, for instance, to shape the evaluation scope by designating clear system boundaries to bound the evaluation, or the concept might be applied within the interpretation of the data, to gain, for example, an understanding of how system elements view the boundaries of their own system. A yellow coding represented a study in which there was some attempt to apply a concept, but it was limited or addressed in an implicit manner. A red color code represented instances in which the concept was not utilized. The aim of this appraisal was not to be overly critical about individual studies but rather to understand the ways in which concepts from systems thinking and complexity science are applied in this body of literature. This process required us to make judgments, and in some instances, the decisions were not necessarily clear cut. In order to increase the validity of this process, 2 reviewers (EM and DM; or EM and ME) independently assessed each study, and disagreements were reconciled through discussion.

Evaluation characteristics

A total of 21 unique evaluations (in 25 separate publications) were identified (see Fig 1). Their characteristics are presented in Table 1, and in-depth descriptions of 2 evaluations, one rooted in systems thinking [41] and another in complexity science [42], are presented in S2 Text. The in-depth descriptions were written to give clear examples of how these approaches have been applied in practice. A range of public health topics were represented in the sample, including social work [43–45], school health [41,46–50], workplace health [51,52], sexual health [48,53,54], health equity policy [55,56], urban planning [57,58], substance use [53,56,59–61], child development [62], public–private partnerships [63], community empowerment and transformation [42,64], and violence prevention [65]. The studies were conducted in 13 countries, which included 9 high-income and 4 middle-income settings: Australia [51,52,57,65], Brazil [55], Chile [62], El Salvador [54], Finland [56], India [62], Israel [43,44], the Netherlands [46,47,58], New Zealand [50], South Africa [48,62], Sweden [59], the United Kingdom [42,49,60,61,63,64,41], and the United States [45,52].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Flow diagram for inclusion of studies.

https://doi.org/10.1371/journal.pmed.1003368.g001

The primary studies in this review were notable for their diversity in terms of the theories and frameworks used to inform the evaluation design and the focus of the analysis. Prominent theories included explicit applications of complexity theory [42,50,60] and diffusion of innovation theory [49]. Studies also used a number of frameworks to structure the analysis and to draw out evaluative findings. This included existing frameworks such as the Cynefin framework [48], Consolidated Framework for Implementation Research [59], a complex adaptive systems framework [54,62], and the socioecological model [41]. Other evaluations featured bespoke frameworks for analysis, including ones that focused on the role of critical events in an intervention’s trajectory [55], a systems framework focusing on governmental subsystems [56], and a framework that was used to identify and categorize different types of “by-effects” or unintended consequences [58].

The process evaluations in this literature base varied in terms of the stage of evaluation planning and conduct in which they drew on complex systems thinking concepts and frameworks. Although the reporting was not always clear, 14 evaluation teams used some facets of systems thinking and complexity science when planning and designing their evaluations [41–47,49,51–53,57,58,60–62,64,65], which ranged from asking systems-oriented research questions to informing the sampling strategy (e.g., a conscious effort to sample different elements or from different levels within the system) and data collection tools (i.e., interview topic guides). Other evaluators used complex systems concepts, theories, or frameworks solely to structure their analyses [48,50,54–56,59,63].

The evaluations identified also drew on a wide range of qualitative methodologies. Ten studies applied a case study design [41–45,50–52,56,60–62,64]. The nature and boundary of a case varied from evaluation to evaluation. Some studies (n = 3), for example, defined a case based on geographical boundaries, and each case represented a geographical locality [42,60,61,64]. Other case study examples included individual families [43,44] or schools [41] or the specific application of a policy [56].

Evaluators utilized a number of different methods for data collection, and 13 applied a mixed methods approach, which included using multiple qualitative data collection methods [41–45,48–50,55–58,62,64]. Seven studies employed a mix of qualitative and quantitative methods [46,47,51–53,59–61,63,65], although all of these studies had substantive qualitative findings. Not all evaluators articulated their rationales for choosing and combining certain qualitative methods, but in general, the different methods were employed to access, understand, and analyze different elements, structures, and relationships within the system. For example, speaking to a range of different actors within the system, through interviews (semi-structured, in-depth, or narrative) and focus groups [41–65], was used to assess different perspectives about an intervention, relationships, and theories of change within the broader system and to make sense of system trajectories. Documentary review and analysis were also relatively common, being used in 7 studies [41,43,44,46,47,50,62–64], and a range of documents were reviewed including media reports, community plans, evaluation documents, and case reports. Documents were used to understand intervention development and implementation and to generate data at different levels within systems, for example, with some evaluators choosing to review national-level documentation and subsequently conduct regional or local-level interviews [41]. Seven of the evaluations identified also conducted both participant and nonparticipant observation, which ranged from observations of meetings to community events [41–44,46,47,49,51,52,64]. In addition to these researcher-led qualitative methods, some evaluators (n = 10) utilized more participatory research techniques, including research seminars and workshops, mapping exercises, the creation of intervention timelines, and other types of group exercises [41,42,48,55,64]. Participatory methods were utilized both as a means of bringing in the perspective of those affected directly by the intervention, as well as a method to check and present interim findings.

Several of the identified process evaluations were conducted alongside or after impact/outcome evaluations of the same intervention. Knai and colleagues integrated data from several evaluative strands including impact and process evaluations [63]. Five studies reported accompanying outcome evaluations, but those results were not presented alongside the process evaluation reports [43,44, 46,47,59,64]. Three studies presented outcome data alongside their process evaluations [50–52, 60,61]. Finally, 2 papers reported independent outcome evaluations that were not linked to their own process evaluations [49,58].

The identified evaluations varied in the extent to which they produced and utilized system maps; 11 produced system maps of some description [41,46–48,51–57,60,63]; of these, only one used a formal system mapping technique: a causal-loop diagram [63]. The other system maps were bespoke maps that depicted different types of logic models [60,63], maps of the system structure [41,53,54], and maps that showed interactions between system elements [51,54,55,57].

Application of concepts from systems thinking and complexity science

Evaluations varied in the extent to which they applied concepts from systems thinking and complexity science to their evaluation design or analysis and concepts from systems thinking were utilized to a far greater extent than complexity concepts. Fig 2 shows this using a traffic light coloring scheme. The figure is structured with different concepts from systems thinking and complexity science in each of the columns. The concepts are presented as belonging along a continuum, with systems thinking on the far left-hand side and complexity science on the far right-hand side. Moving along the spectrum, from systems thinking to complexity science, represents a movement from static to dynamic. Key systems thinking concepts, on the left-hand side of the figure, are the structure of a system, its elements, and the relationships between them. Utilizing these allows researchers to create relatively static depictions of a system. Moving toward the middle of the figure, concepts from complexity science are introduced, which include attributes and dimensions of an intervention, and then a system undergoing change. The far right-hand side of the figure includes concepts that feature within the complexity sciences to computationally model complex systems in order to simulate and predict behavior and outcomes and to understand an evolving system.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Included studies and the degree to which they apply concepts from systems thinking and complexity science.

Each color-coded circle denotes the degree to which an evaluation applied the associated concept to any stage of the evaluation process. Green: study explicitly applied the concept; yellow: study attempted, or implicitly applied the concept; red: concept was not applied.

https://doi.org/10.1371/journal.pmed.1003368.g002

The evaluations identified in this review consistently applied key concepts from systems thinking: the identification and description of the system structure, including the different system elements and their differing perspectives. Thinking systemically also means making sense of the boundaries of a system and making decisions about what constitutes “the system” and what might be considered within or outside of the system. Although system maps are not a necessary element of systems thinking, they can be helpful for making sense of and depicting system boundaries, as articulated by both those acting within the system (“first-order” boundary judgments) and those studying it (“second-order” boundary judgments) [66]. Few evaluations (n = 3) in the sample [42,45,64] had explicit discussions of boundaries and the ways in which, or indeed if, boundary judgments were made. By contrast, 11 studies produced some form of system diagram [41,46–48,51–57,60,63], implying that boundary judgments were likely at least implicitly considered by evaluators. The identified papers focused analytically on the relationships between systems elements. Such a focus is understandable and indeed, a pre-requisite for being labeled as a system approach; without a focus on relationships and interactions—the key tenet of systems thinking—the approach fails to be systemic.

Somewhat surprisingly, only 4 fewer evaluations explicitly utilized a range of complexity concepts to assess changes within the system resulting from, or co-occurring with, intervention implementation over time [42,46,47,50,55]. By their nature, public health problems and the systems in which they are created and shaped are complex [40], and as a result, we might expect to see a more explicit attempt to use complexity concepts to generate evidence on public health interventions. Complexity science introduces a number of additional concepts that may be of value to researchers who seek to evaluate the mechanisms by which public health interventions have impacts in real-world environments. These concepts are used to describe, analyze, measure, and estimate attributes of change. The change first occurs within and across the system elements, and these collective changes result in emergent system change.

In the body of literature identified in this review, concepts from the complexity sciences, such as those that are used to understand change within systems, were utilized less frequently compared with concepts that could be used to describe static “snapshots” of systems. Although some papers were notable for applying a number of complexity concepts [42,46,47,50,55], the majority drew on only a few complexity-informed concepts in order to describe key mechanisms that might drive system change, such as a feedback loop. Researchers did not always provide a rationale for how the concepts had been chosen or specifically considered within the context of data collection and analysis. An exception to this was one study that created an explicit analytic framework to identify and explain a range of by-effects (unintended consequences stemming from an intervention) [58]. The framework categorized policy achievements as foreseen or unforeseen and desired or undesired [58]. Within the evaluations identified, the complexity concepts that were most frequently used included nonlinearity, feedback, and adaptation.

Phase 1: A static system description

In the first part of this 2-phase framework, we propose that evaluators conduct a period of research in order to gain an initial understanding of the system, including the system structure, the boundaries, the constituent elements, and the relationships between these [6,14] at a given time point [24]. This description represents a snapshot of the system at one point in time. For many evaluators, it may make sense to capture the “initial conditions” or “initial state” of the system at the time the intervention is first implemented. In these cases, the evaluation would involve a period of familiarization and the first part of data collection as the intervention is being implemented or shortly thereafter. In this stage, evaluators would also begin to hypothesize some of the ways that the intervention may lead to change within the system (which may be informed by the intervention’s theory of change, if one is articulated). If the intervention designers have not described a theory of change, evaluators at this stage should articulate one by mapping out the initial hypotheses of system change.

In Phase 1, evaluators would begin to make sense of and document the “local rules” that govern both the intervention and the system, including the rules that govern how different system elements interact and relate to each other and how the intervention operates and relates to different parts of the system. In undertaking Phase 1, evaluators would draw on concepts that are most closely aligned with systems thinking (the left-hand side of Fig 2 and first half of Table 2) and use these to structure the initial data collection and analysis. Following the identification of the system structure, elements, boundaries, and relationships, evaluators should begin to consider some of the ways in which the intervention may lead to changes within the system. Evaluators could ask how the system elements respond to the intervention, comparing different stakeholder perspectives. Evaluators could also begin to assess system coherence by analyzing the degree to which the intervention is aligned with the interests of those in the system or the instances in which the intervention may “swim against the tide” [72,73].

In Phase 1, data should be collected from a range of different actors within the system. Evaluators may find a number of different data collection methods useful, including, but not limited to, an initial documentary review, interviews, and workshops. The boundary decision and the identification of system elements will inform from whom data are collected and through which methods [14].

As part of this process and as a way of analyzing the data collected in Phase 1, it may be helpful to create a map of the system. The type of map created will depend on the role it is to play in the evaluation. For example, if a map is made to visually represent the system structure and boundaries to help depict and understand the system structure and relationships between the system elements [57], it may be created through a semi-structured brainstorming session or interviews and the analysis of the data collected in Phase 1. Alternatively, evaluators may choose to create more structured system maps, drawing on established mapping methods, such as concept mapping or group model building, in order to map out causal linkages between system variables [74]. In these instances, Phase 1 represents an opportunity for initial preparatory work for the map creation process.

The output of Phase 1 would be relatively descriptive and static: a qualitative description of the system structure, elements, boundaries, and relationships which may well be depicted on a map, as well as some hypotheses about how the intervention may lead to system change, including the ways in which the elements and the system as a whole adapt and co-evolve in response. The hypotheses of system change may be depicted as a theory of change, which maps out how the intervention could lead to impacts, with particular consideration given to the pathways and mechanisms by which that change is brought about [6]. The initial system description and possible pathways for system change would then inform Phase 2.

Phase 2: A system undergoing change

The second phase of evaluation would examine emergent properties of the system and explore system change stemming from the intervention, drawing on a complexity perspective. In Phase 2, evaluators should be prepared to follow the pathway of emergent findings. In this sense, the evaluation needs to be adaptable, flexible, agile, incorporate multiple perspectives, and deal with uncertainty to support real-time decision-making. Evaluators would use the data collected in Phase 1 (particularly the emerging hypotheses about system change) to develop specific research questions about the intervention and the system. In defining the research questions, there is an opportunity to explicitly apply some of the complexity concepts—for example, by asking questions about the adaptive responses within different elements of the system, unintended consequences of the intervention for different population groups, or emergent system outcomes as the system co-evolves with its broader environment. It is not our suggestion that evaluators attempt to apply all complexity concepts to any one evaluation but rather focus on those that can generate useful evidence for decision-making [71]. Although the timing of Phase 2 may be determined by the theory of change, it may also be influenced by the timing of other types of data collection. For example, the process evaluation may accompany an impact evaluation that prespecifies time points for data collection [16,17].

At this stage, a more formal period of sampling and data collection would begin, to complement data collected in Phase 1 and to focus the sampling and data collection strategies to better answer the research questions. The specific sampling strategy and data collection methods will vary from evaluation to evaluation, but any process evaluation applying a complex systems perspective would sample multiple types of participants (e.g., different system elements) and use multiple methods [6,66]. As the papers in this review underscore, the careful use and reporting of different qualitative methods underpinned by complex systems theoretical principles can help an evaluator assess different perspectives across and within system levels, as well as different types of information [27]. Analyzing data generated through different qualitative methods can be used to bring a dynamic component to the evaluative research; for example, documents can be used to understand previous decisions and interviews or observations could then be used to understand the trajectory of those decisions and their impact across the system on different population groups [27]. Evaluators should consider the timing and ordering of mixed methods; a document review might, for example, provide important context in order to inform interview schedules [27]. Complexity concepts have traditionally been used within the context of quantitative and modeling methods. However, we argue that there is no reason that these concepts should not be of interest within a process evaluation using qualitative methods, particularly as many deal specifically with system changes upon which qualitative research could shed light [41,48].

During the analysis stage, the evaluators would begin to make sense of the emerging findings through the application of relevant complexity concepts. For example, an evaluation concerned with understanding the ways in which the intervention may lead to the amplification or dampening down of certain kinds of systemic change would have an explicit focus on identifying feedback loops within the system [75], or it might make sense (based on hypotheses generated in Phase 1) to focus the analysis on understanding how the system’s history influences its trajectory and adaption in response to the introduction of an intervention [76]. As the analysis is undertaken, there is likely a need to collect more data, in a kind of evaluative feedback loop. Such a process will be familiar to those who apply iterative research designs [17,77]. Throughout the analysis, evaluators would revisit, revise, and refine the theory of change and system map in light of the new data.

Generating outputs can be a challenge for public health evaluators applying a systems perspective. It is difficult to convey complex findings in a manner that is useful and timely for decision makers and does not result in an overly reductionist account or a confusingly “complex” set of findings. This is particularly a concern for qualitative research in which large volumes of data are collected. We suggest that one way to present the findings from a complex systems process evaluation is to create a “system story,” wherein the evaluator describes and analyses how the intervention embeds and co-evolves with the system and its elements overtime [3].

A more traditional approach to process evaluation is often rooted in the intervention itself, rather than the system in which that intervention is implemented. As a result of this orientation, such an evaluation generally considers the intervention and its immediate implementation processes and mechanisms, although there may be some consideration of more distal mechanisms and impacts [17]. In addition, more traditional process evaluations tend to adhere to research protocols that may themselves be relatively inflexible. A process evaluation from a complex systems perspective takes the system as the initial starting point of the analysis and considers the ways in which the intervention may lead to immediate, as well as more distal impacts, and the ways in which that intervention may change how the system elements—and the system as a whole—behave. Doing so will inherently require a flexible, adaptive, and iterative design. The framework presented here suggests at least 2 phases of data collection, with the understanding that the second phase will likely include an iterative process of defining research questions and collecting and analyzing data. Utilizing a longitudinal design with data collected over a relatively lengthy period of time or at more than one time point in order to capture a dynamic system undergoing change [24,67,71] may be a challenge to public health evaluators because it implies longer timescales [78], a move away from more standard evaluative approaches and a degree of risk with which some funders and decision makers may be uncomfortable. In addition, it may challenge traditional public health evaluation methods that strictly follow protocols in an attempt to control for internal validity [16]. In contrast, a complex systems approach to evaluation must inherently plan to adapt and change in response to early evaluative findings, as well as in response to the changing intervention and broader system. As a result of an adaptive evaluation design, the distinction between different types of evaluation (such as formative, process, outcome, and impact) may be less clearly defined. As evaluators follow the pathways of emergent hypotheses and findings, it may well make sense to, for example, measure or predict impacts alongside process mechanisms. Finally, further work remains on the ways in which realist and mixed methods approaches can more explicitly contribute to a process evaluation from a complex systems perspective, but it is beyond the scope of this current review.

Limitations

The nature of the review topic area required the research team to make a number of judgments throughout the review process. First, judgments were made regarding which studies to include or exclude on the basis of their public health relevance and the degree to which they featured a complex systems perspective. Although the majority of decisions were clear cut, the reviewers, in discussion with one another, had to make judgments in cases that were less obvious, and there is the possibility that other review teams would have made different decisions. In addition, there was a subjective element in deciding which concepts from systems thinking and complexity science to highlight; we sought to capture the key principles associated with each of the traditions with the goal of this list being used by those wishing to draw on systems thinking and complexity science within the context of public health evaluation. We recognize that other reviewers might have chosen to highlight other concepts. Finally, the critical appraisal of the studies again required judgments. In order to increase validity, 2 reviewers completed the process independently and reconciled their decisions, but the decisions were not always clear cut.

Another limitation of this review is the focus on studies which self-identify as taking a systems and/or complexity-informed approach. This focus has 2 possible limitations: First, it excludes studies that may be compatible with systems thinking but do not cite systems literature or draw explicitly on systems concepts, and second, it may include studies that utilize the terminology of complex systems, because it has become somewhat fashionable in the last few years, but fail to apply the concepts in such a manner that investigates complex uncertainties to generate better evidence for decision-making [71]. Taking the first concern, many rigorous qualitative studies foreground context in their research focus and analyses, considering the broader economic, social, political, cultural, environmental, and historical factors that impact interventions’ trajectories and influence diverse population groups [79]. As we have contended, “system” and “context” are broadly synonymous, in that all of a system can arguably be considered “contextual.” Therefore, qualitative research that actively engages with the broader context may apply a perspective that is compatible with systems thinking, without using the accompanying systems terminology. Indeed, the MRC Guidance on “Process Evaluation of Complex Interventions,” had limited reference to complex systems theory and terminology but nevertheless advocated a systems-compatible approach to process evaluation, namely, an approach that explores the “dynamic relationships between implementation, mechanisms and context, the importance of understanding the temporally situated nature of process data in understanding the evolution of an intervention within its system” [17,71]. With regards to the second concern, complex systems thinking is currently in vogue in public health, which can be seen in the growth of calls for the application of a complex systems perspective to public health practice and research [1,35,80,81]. Although many researchers are grappling with how to harness insights from the systems thinking and complexity science traditions to improve public health research, there is some concern that complex systems literature and concepts have been used without researchers truly engaging with the underlying theory [71]. These limitations suggest a number of opportunities for further research in this field. In particular, future research could fruitfully explore the degree to which public health literature—on intervention development and evaluation—is compatible with a complex systems perspective, even when not explicitly described as such. Other research might identify process evaluations that do not explicitly adopt a complex systems approach and analyze the added value of an explicit engagement with the systems and complexity literature.

Finally, we limited our search to English-language publications and relied on 2 previous reviews and an expert consultation to identify qualitative process evaluations from a complex systems perspective that were published prior to 2014, which is a limitation of our search’s sensitivity. The studies identified through these means may have been influenced by other researchers’ interpretations and possible biases. Any papers not identified from our search may have potentially added further to our methodological synthesis and the recommendations we put forward in the Discussion.