https://orcid.org/0000-0001-7318-3598
Figures
Abstract
Background
Implementing evidence into clinical practice is a key focus of healthcare improvements to reduce unwarranted variation. Dissemination of evidence-based recommendations and knowledge brokering have emerged as potential strategies to achieve evidence implementation by influencing resource allocation decisions. The aim of this study was to determine the effectiveness of these two research implementation strategies to facilitate evidence-informed healthcare management decisions for the provision of inpatient weekend allied health services.
Methods and findings
This multicentre, single-blinded (data collection and analysis), three-group parallel cluster randomised controlled trial with concealed allocation was conducted in Australian and New Zealand hospitals between February 2018 and January 2020. Clustering and randomisation took place at the organisation level where weekend allied health staffing decisions were made (e.g., network of hospitals or single hospital). Hospital wards were nested within these decision-making structures. Three conditions were compared over a 12-month period: (1) usual practice waitlist control; (2) dissemination of written evidence-based practice recommendations; and (3) access to a webinar-based knowledge broker in addition to the recommendations. The primary outcome was the alignment of weekend allied health provision with practice recommendations at the cluster and ward levels, addressing the adoption, penetration, and fidelity to the recommendations. The secondary outcome was mean hospital length of stay at the ward level. Outcomes were collected at baseline and 12 months later. A total of 45 clusters (n = 833 wards) were randomised to either control (n = 15), recommendation (n = 16), or knowledge broker (n = 14) conditions. Four (9%) did not provide follow-up data, and no adverse events were recorded. No significant effect was found with either implementation strategy for the primary outcome at the cluster level (recommendation versus control β 18.11 [95% CI −8,721.81 to 8,758.02] p = 0.997; knowledge broker versus control β 1.24 [95% CI −6,992.60 to 6,995.07] p = 1.000; recommendation versus knowledge broker β −9.12 [95% CI −3,878.39 to 3,860.16] p = 0.996) or ward level (recommendation versus control β 0.01 [95% CI 0.74 to 0.75] p = 0.983; knowledge broker versus control β −0.12 [95% CI −0.54 to 0.30] p = 0.581; recommendation versus knowledge broker β −0.19 [−1.04 to 0.65] p = 0.651). There was no significant effect between strategies for the secondary outcome at ward level (recommendation versus control β 2.19 [95% CI −1.36 to 5.74] p = 0.219; knowledge broker versus control β −0.55 [95% CI −1.16 to 0.06] p = 0.075; recommendation versus knowledge broker β −3.75 [95% CI −8.33 to 0.82] p = 0.102). None of the control or knowledge broker clusters transitioned to partial or full alignment with the recommendations. Three (20%) of the clusters who only received the written recommendations transitioned from nonalignment to partial alignment. Limitations include underpowering at the cluster level sample due to the grouping of multiple geographically distinct hospitals to avoid contamination.
Conclusions
Owing to a lack of power at the cluster level, this trial was unable to identify a difference between the knowledge broker strategy and dissemination of recommendations compared with usual practice for the promotion of evidence-informed resource allocation to inpatient weekend allied health services. Future research is needed to determine the interactions between different implementation strategies and healthcare contexts when translating evidence into healthcare practice.
Author summary
Why was this study done?
- Healthcare delivery does not always reflect the most up-to-date research evidence.
- There are high levels of evidence to suggest that inpatient allied health services provided during weekends achieve greatest benefits in subacute rehabilitation wards.
- Most weekend allied health services are provided to acute general medical and surgical wards, where there is uncertain evidence of impact.
- Translation of evidence into practice is constrained by a limited understanding of which implementation strategies are most effective for specific settings.
What did the researchers do and find?
- We conducted a cluster randomised controlled implementation trial to compare the effectiveness of two research implementation strategies across 132 hospitals in Australia and New Zealand.
- We provided hospital managers with either evidence-based weekend allied health practice recommendations or access to a knowledge broker in addition to the recommendations, over a 12-month period.
- Neither implementation strategy was able to be shown effective for ensuring better alignment of weekend allied health provision with practice recommendations; no impacts on hospital length of stay were identified.
What do the findings mean?
- Evidence dissemination and knowledge brokering are thought to facilitate the translation of research evidence into practice.
- Our study was unable to find whether either of these strategies substantially influenced weekend allied health service decision-making by hospital managers.
- It is possible to study the impact of research implementation using robust trial designs; however, challenges achieving adequate statistical power are a barrier to these evaluations.
Citation: Sarkies MN, Robins LM, Jepson M, Williams CM, Taylor NF, O’Brien L, et al. (2021) Effectiveness of knowledge brokering and recommendation dissemination for influencing healthcare resource allocation decisions: A cluster randomised controlled implementation trial. PLoS Med 18(10): e1003833. https://doi.org/10.1371/journal.pmed.1003833
Academic Editor: Elvin Hsing Geng, Washington University in St Louis, UNITED STATES
Received: March 11, 2021; Accepted: October 4, 2021; Published: October 22, 2021
Copyright: © 2021 Sarkies et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Individual participant demographic data cannot be shared publicly because of the potential risk for re-identification. Cluster-level and ward-level data are available within the manuscript and its Supporting information files.
Funding: This work was funded by the National Health and Medical Research Council (NHMRC) Australia (APP1114210); https://www.nhmrc.gov.au/ to TPH. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Healthcare systems worldwide continue to grapple with unwarranted variation in quality and safety. On average, an estimated 60% of care is delivered according to recommended guidelines [1–4], while up to 30% is considered low-value care [5–7]. Underuse of effective treatments and overuse of those with questionable benefit are arguably responsible for substantial inefficiency and lost opportunity to improve patient outcomes, particularly when considering improvements that could be realised by reallocating resources to high-value care.
The evidence-to-practice gap often manifests through decisions and negotiations within and across multiple levels of the health system [8]. These phenomena create a complex ecosystem, where the delivery of care can be dependent upon policy and managerial decisions regarding the organisation of resources. One highly topical area of healthcare policy and management decision-making is the provision of weekend allied health services to inpatient wards [9]. Internationally, there is substantial variation in access to allied health professionals within hospitals (e.g., physiotherapists, occupational therapists, speech pathologists dietitians, social workers, and podiatrists) [10–14]. This variability is typified by service provision during weekends, where some hospitals extend limited provision of allied health services [15,16]. A recent systematic review and meta-analysis reported that additional weekend allied health services in subacute rehabilitation units can reduce hospital length of stay by over two days and is a cost-effective approach to improve function and health-related quality of life [17]. However, the benefits were less clear for acute general medical and surgical wards. Despite this evidence, subacute rehabilitation units often provide less allied health during weekends, compared to acute general medical and surgical wards, both in absolute and relative terms [11].
Accessing and applying research evidence to guide resource allocation decisions can be difficult for healthcare managers [18,19]. Therefore, it is imperative to evaluate strategies that promote evidence-informed decision-making to ensure that the benefits of research on weekend allied health service models can be translated to improved health outcomes. Guidelines and recommendations are widely used to disseminate concise instructions regarding patient care. Recommendations have been found to increase awareness of key messages, change attitudes and knowledge [20–22], and impact practice in some circumstances [23–26]. However, they do not always lead to meaningful changes in behaviour [27–30]. Given the feasibility and low cost involved, dissemination of evidence-based practice recommendations may be an efficient way to change practice under certain conditions, even if it is less effective than more interactive alternatives. A more interactive, and resource-intensive, approach is the use of knowledge brokering to support dissemination and implementation of recommendations into evidence-informed decision-making [31–36]. Knowledge brokers are intermediary agents who build relationships between decision-makers and researchers, by sharing expert knowledge and establishing communication channels [37]. Much of this work occurs informally [38]. Yet, these roles are increasingly being formalised and institutionalised [39,40], despite limited evidence to support their effectiveness [36,41]. The substantial cost and resources required to deliver formalised knowledge broker roles [42,43] require evidence of both effectiveness and cost-effectiveness to justify investment. This is because the health system is characterised by finite funding, which must be allocated to competing needs: providing funds for activities with unknown levels of effectiveness such as knowledge brokering is difficult to justify, as those same funds can no longer be used for alternative activities that are known to effectively improve health outcomes elsewhere in the health system.
The aim of this study was to determine the effectiveness of knowledge brokering and dissemination of evidence-based practice recommendations for weekend allied health resource allocation decisions by hospital managers.
Methods
Monash Health Research Ethics Committee approved this research (HREC/17/MonH/44). The study protocol has been published (S1 Text) [44] and registered (Australian New Zealand Clinical Trials Registry ACTRN12618000029291); Universal Trial Number (UTN): U1111-1205-2621.
Context
Allied health professionals routinely deliver inpatient services Monday to Friday for hospitals in high-income countries. In certain parts of the world, these services are also extended during Saturday and Sunday, with Saturday physiotherapy services being the most provided [10,45]. In Australia, approximately 60% of acute hospital wards and 30% of subacute wards provide physiotherapy during weekends [11], which contrasts with Level-I evidence indicating that the benefits are clearer in subacute rehabilitation units [17]. A recent study by Haines and colleagues demonstrated that weekend allied health services could be removed from acute general medical and surgical wards without impacting health or service delivery outcomes and that redesign and reinstatement of these services also did not change these outcomes [16].
Trial design
In this multicentre study, we conducted a blinded (data collection and analysis) three-group, parallel cluster randomised controlled trial with concealed allocation to compare two alternate research implementation strategies with a control. Clustering occurred at the organisation level where weekend allied health staffing decisions were made within each healthcare organisation to avoid the potential risk of contamination between units of randomisation (e.g., network of hospitals or single hospital). Hospital wards were nested within these decision-making structures. For example, organisations made up of geographically distinct hospitals that made independent decisions in relation to allied health staffing were randomised as separate clusters; those making decisions across hospitals within the organisation were randomised as a single cluster. The diversity in decision-making structures across Australian and New Zealand hospitals constrained the ability to prespecify the number of potentially eligible hospitals/wards to be included within each cluster. Randomisation was stratified based on self-reported geographical classification, as either metropolitan or rural (including regional and remote). This study design is considered the most suitable to address questions of effectiveness, avoid potential contamination across study conditions, and capture outcomes at the system levels where changes were expected to occur.
During the conduct of this study, there was one randomisation error and one modification to the statistical methods described in our study protocol. The randomisation error occurred when one cluster was randomised to the recommendation group but mistakenly did not receive it because of human error. Data were still collected from this cluster and analysed according to the group to which they were assigned. Modifications to the analysis are outlined in the statistical methods section. A CONSORT Extension for Cluster Trials checklist for this study is provided in S2 Text, and the trial was also reported according to the Standards for Reporting Implementation Studies Statement (StaRI) in S3 Text.
Participants and setting
This study took place across a sample of Australian and New Zealand hospitals. Eligible hospitals were those providing acute or subacute services, with either public or private funding arrangements. Specific wards of interest were general medical and surgical and subacute rehabilitation. Specialist hospitals, including maternity, paediatric, cancer, mental health, and palliative care, were excluded, as no research regarding weekend allied health provision had been identified in these settings. Hospital managers who were responsible for inpatient weekend allied health resource allocation decisions at each cluster were eligible to receive the interventions on behalf of the cluster after providing written informed consent.
Interventions
A detailed description of the three study conditions according to the Template for Intervention Description and Replication (TIDieR) guidelines [46] is provided in the published protocol [44], and specification of the implementation strategies delivered to the study groups is reported in Table 1 [47]. Each strategy was commenced at the time of randomisation for a period of 12 months to implement specific recommendations regarding weekend allied health provision, derived from a systematic review and meta-analysis [17] and summarised in Box 1. The full evidence-based practice recommendations are provided (S4 Text).
Box 1. Summary of evidence-based policy recommendations for weekend allied health provision
- Reduce weekend allied health staffing to a criterion of clinical priorities and exceptions that ensure between 0% and 0.1% of total allied health service events on acute general medical and surgical wards are delivered on weekends.
- Increase weekend allied health staffing for subacute rehabilitation units to provide physiotherapy or a combination of physiotherapy and occupational therapy on Saturdays and physiotherapy on a case-by-case basis for stroke patients and, occasionally, to other patients when a clear need is evident on Sundays. This would ensure that between 10% and 20% of total service events for these professions are provided on weekends.
Briefly, three study conditions were delivered at the cluster level: (1) usual practice wait list control; (2) dissemination of written evidence-based practice recommendations; and (3) access to a webinar-based knowledge broker in addition to the recommendations. The waitlist control group experienced usual practice conditions according to their local setting. Upon study completion, they received the evidence-based recommendations for weekend allied health provision. Participants in the recommendation group were provided with an evidence-based weekend allied health practice recommendation (detailed and summarised) document via email. This document contained specific recommendations for the proportion of total allied health services that should be delivered during weekends. The document was constructed with an outline of key messages, executive summary, and presentation of the full research methods and findings [48]. The knowledge broker group was provided the same recommendation document and additional access to a knowledge broker who facilitated the transfer of relevant information to promote evidence-informed decision-making. The knowledge broker offered support via interactive online webinar, telephone, or email, in one-on-one and group settings. Prompting questions by the knowledge broker were informed by the COM-B (capability, opportunity, motivation, and behaviour) behaviour change model [49]. This support included individual needs assessments and developing a 12-month plan to address the recommendations. These supports followed an iterative process depending on participant needs, based on factors perceived to be associated with effective strategies from a recent systematic review [41]. The dosage and duration of the interventions were based on a similar study evaluating a knowledge broker role [36], which aligned with our hypothesis that a 12-month intervention duration would prove sufficient time for hospital managers to develop and implement a business case for change.
Outcomes
The primary outcome of interest was whether weekend allied health service provision at both the cluster and ward level aligned with the recommendations at 12-month follow-up. This outcome addressed the adoption of evidence-based practice recommendation, penetration among eligible hospital wards, and fidelity to the recommendation. Alignment with the recommendations was determined according to the number of allied health service events occurring during weekends, as a proportion of the total allied health service events for the cluster or ward, over a one-month period. A ratio of allied health full-time equivalent staffing was used, where service event data were not available (n = 3 clusters). Allied health service event data were collected at the time of randomisation for the preceding calendar month and the same calendar month, 12 months later. For cluster-level analysis, each cluster received a single classification as either (1) fully aligned with the recommendations for both acute wards and subacute units; (2) partially aligned with the recommendations (if acute wards are aligned but subacute units are not aligned, or vice versa); or (3) not aligned with policy recommendations. For the ward-level analysis, each acute ward or subacute unit received a single classification as either (1) fully aligned with the recommendations; or (2) not aligned with policy recommendations.
The secondary outcome was the mean hospital length of stay at each ward for the calendar month 12 months after study entry. We also collected equivalent data for each ward from the same calendar month 12 months earlier. Hospital length of stay was extracted from administrative data sources [54]. Detailed process and economic outcomes described in our protocol are planned for other publications.
Randomisation
Study investigators consulted with each healthcare organisation to determine their decision-making structure for allied health staffing to reduce the risk of contamination between study groups. Healthcare organisations that made allied health staffing decisions across multiple hospitals were treated as a single unit of recruitment and randomisation, where hospitals with independent decision-making processes within a broader healthcare organisation were treated as separate units. Hospital wards were nested within clusters and randomised using a random number sequence, generated by a single investigator (TPH) using an online software application [55] with permuted blocks within randomisation strata of sizes of 3, 6, or 9. Investigators conducting recruitment, data collection, and analysis (MS and MJ) were blinded, as per procedures outlined in the published protocol [44].
Statistical methods
Initially, the use of the nonparametric rank-based classical hypothesis test was planned to compare data for the primary outcome. However, changes were made to account for the observation that several clusters and wards were already aligned with the evidence-based recommendations. We changed our analysis approach to an ordered logit regression at the cluster level and logistic regression analysis at the ward level, so we could statistically adjust for the baseline status of each unit of analysis when comparing 12-month follow-up alignment with the recommendations between groups. These ANCOVA style analyses are aligned with recommendations from the Cochrane Handbook for Systematic Reviews of Interventions [56] for how baseline values can be appropriately incorporated into an interventional analysis framework. We used robust variance estimates to account for dependency of clustering across wards that had the same decision-makers, such that this analysis was still analysing data relative to the level of randomisation. The mean hospital length of stay was compared between groups in a ward-level analysis. Linear regression using baseline mean hospital length of stay as a covariate and robust variance estimates at the level of the decision maker was conducted.
Analysis of cluster-level and ward-level data was undertaken according to the group to which they were assigned by an analyst (MS) blinded to group allocation, using three mock codes representing different sequence allocation patterns. Multiple imputation using chained equations was performed to impute 50 datasets for missing values in both baseline and follow-up alignment with the recommendations. Geographical classification was used as the independent variable when imputing missing data at the cluster level; geographical classification and ward classification were used at the ward level. Two sensitivity analyses were conducted: The first was a complete case analysis for comparison with the analysis of imputed data, and the second was a multivariable analysis to adjust for other potential baseline confounders (cluster level: geography and full-time research staffing; ward level: geography and ward type). All analyses were adjusted for the number of weekday days and weekend days within the month to ensure that calendar years changes in the proportion of day types did not impact the results. Analyses were undertaken using Stata version 13.1 (StataCorp, College Station, Texas). Sample size estimates are reported in our study protocol [44].
We planned two levels of analysis, one to be conducted at the cluster level (one unit of data per cluster) and the other to be conducted at the ward level (nested within cluster). The study sample size was determined based on the most conservative unit of assessment for our primary outcome at the cluster level (organisation-level where weekend allied health staffing decisions were made), as an adequate sample size at the cluster level would also prove sufficient at the ward level. Further power analysis at the ward level was not conducted because there was no reasonable way to estimate the likely intracluster correlation coefficient (ICC) for this healthcare context and our main sample size constraints pertained to practicality of recruitment at the cluster level. A sample size of 25 clusters per group was estimated to provide greater than 80% power, assuming that 50% of clusters in either intervention group and 10% in the control group completely aligned with the policy recommendations. The diversity in decision-making structures across Australian and New Zealand hospitals constrained our ability to prespecify the number of hospitals/wards within each cluster and recruit sufficient clusters per group, as often allied health staffing decisions were made across hospitals/wards within organisations resulting in fewer potential units of randomisation. Study data used in these analyses are provided (S1 and S2 Data).
Results
The first cluster was recruited on 7 February 2018, and the final cluster completed their intervention in January 2020; final data collected for that period on 30 April 2020. A total of 45 clusters were randomised to either the control (n = 15), recommendation (n = 16), or knowledge broker (n = 14) groups. There was one cluster that did not provide any outcome data and three that did not provide follow-up outcome data (recommendation n = 2; knowledge broker n = 2) leaving a rate of 9% loss to follow-up (Fig 1). No adverse events were recorded. Within the clusters, there were n = 132 hospitals, n = 833 wards, and n = 204 hospital managers, whose baseline characteristics are provided in Tables 2 and 3.
One cluster did not provide baseline or follow-up data; three clusters did not provide follow-up data only.
The summative raw data for the primary and secondary outcomes are presented in Tables 4 and 5. Proportion of total allied health service events provided on weekends is presented for each group, along with the corresponding number and percentage of clusters or wards aligned with the recommendations. None of the clusters in the control or knowledge broker groups transitioned from “not aligned” at baseline to “partial” or “full alignment” at 12-month follow-up. Three clusters (21%) that did not align to the practice recommendations, from the written recommendation document only group, transitioned to partial alignment within the study period. The flow of cluster alignment from baseline to follow-up, by implementation strategy group, is presented in Figs 2–4 [57]. The most common reasons for missing follow-up data across all three groups were (1) that a different contact person was used (due to staff turnover) who did not know how to extract all the required data; and (2) the contact person did not have time to extract all the required data.
The effect size estimates for the implementation strategies compared to the control are presented in Table 6. Adjusted for baseline recommendation alignment, there was no significant difference for the primary outcome of alignment with the recommendations between the groups at the cluster level using ordered logit regression (recommendation versus control β 18.11 [95% CI −8,721.81 to 8,758.02] p = 0.997; knowledge broker versus control β 1.24 [95% CI −6,992.60 to 6,995.07] p = 1.000; recommendation versus knowledge broker β −9.12 [95% CI −3,878.39 to 3,860.16] p = 0.996). These findings were similar with a multivariable sensitivity analysis that included potential confounding baseline variables: geographic location and whether a full-time allied health researcher was employed (recommendation versus control β 19.65 [95% CI −18,116.13 to 18,155.43] p = 0.998; knowledge broker versus control β 1.93 [95% CI −22,189.52 to 22,193.38] p = 1.000; recommendation versus knowledge broker β −9.28 [95% CI −4,601.58 to 4,583.01] p = 0.997). There was no significant difference for the primary outcome of alignment with the recommendations between the groups at the ward level using logistic regression (recommendation versus control β 0.01 [95% CI 0.74 to 0.75] p = 0.983; knowledge broker versus control β −0.12 [95% CI −0.54 to 0.30] p = 0. 581; recommendation versus knowledge broker β −0.19 [−1.04 to 0.65] p = 0.651). These findings were similar with a multivariable sensitivity analysis that included potential confounding baseline variables: geographic location and ward type (recommendation versus control β 0.42 [95% CI −0.62 to 1.46] p = 0.430; knowledge broker versus control β 0.15 [95% CI −0.47 to 0.76] p = 0.639; recommendation versus knowledge broker β 0.04 [95% CI −0.96 to 0.87] p = 0.925).
For the secondary outcome of hospital length of stay, there was no significant difference between the groups at the ward level using linear regression (recommendation versus control β 2.19 [95% CI −1.36 to 5.74] p = 0.219; knowledge broker versus control β −0.55 [95% CI −1.16 to 0.06] p = 0.075; recommendation versus knowledge broker β −3.75 [95% CI −8.33 to 0.82] p = 0.102. These findings were similar with a multivariable sensitivity analysis that included potential confounding baseline variables: geographic location and ward type (recommendation versus control β 1.92 [95% CI −1.85 to 5.70] p = 0.308; recommendation versus knowledge broker β −5.32 [95% CI −13.95 to 3.31] p = 0.213), although a significant difference was identified for the knowledge broker versus control (β −0.71 [95% CI −1.38 to −0.05] p = 0.037).
The primary and secondary outcomes, at the cluster or ward level, were largely unaffected by a complete case sensitivity analysis presented in Table 7.
Process outcomes
The knowledge broker strategy dose varied from that specified in the protocol across sites, due to differing levels of participant engagement. There were 17 interactive online knowledge broker support webinars (duration one to two hours) conducted in total to facilitate the transfer of relevant information to promote evidence-informed decision-making. Healthcare decision maker participation was higher in the initial (range 6 to 18) compared to follow-up (range 2 to 8) webinars. Typically, the first webinars were attended by representatives from each allied health profession; however, the second webinar was predominantly attended by decision-makers from the physiotherapy and occupational therapy professions because the evidence and recommendations mostly pertained to these roles. Only two clusters (14%) participated in more than two webinars with the knowledge broker. A desire to make an internal decision regarding weekend allied health service provision was the most common reason for nonparticipation. Two clusters (14%) did not engage with the knowledge broker without providing a reason. Further detail regarding the knowledge broker webinar support sessions is published elsewhere [58]. Other process, economic, and qualitative outcomes specified in our protocol are planned to be reported in other publications.
Discussion
This study was unable to identify differences between the knowledge broker and dissemination of written evidence-based practice recommendation strategies employed for hospital managers to improve the alignment of weekend allied health services with current evidence. These findings were possibly driven by the high rate of baseline alignment with the recommendations in the knowledge broker group and inadequate statistical power at the cluster-level analysis, despite including 132 hospitals and 833 wards. All 15 clusters assigned to the control group continued providing weekend allied health that was not aligned with the recommendations. Three clusters (21%) from the dissemination of written recommendations only group transitioned from nonalignment at baseline to partial alignment 12 months later. None of the clusters who received the knowledge broker in addition to the recommendations transitioned to partial or full alignment with the recommendations. The reduction in hospital length of stay for the knowledge broker group compared to the control indicates that there may have been an effect but the precise mechanism is unclear without an observed change in recommendation alignment.
Our study results are concordant with the only other three-arm trial published on the dissemination of recommendations and knowledge brokering for implementing evidence into healthcare policies and programs [36]. This earlier research was also unable to demonstrate a positive effect of their knowledge broker implementation strategy; however, the authors reported that impacts may have been more pronounced for health departments with low baseline organisational research culture. Achieving our desired outcomes at only a few organisations aligns with this, and other previous research indicating the effect of research implementation strategies may be contextually specific to organisational characteristics, such as strategic priority, leadership, and readiness [59,60]. The knowledge broker group in our study was characterised by higher baseline rates of recommendation alignment, despite pilot work conducted prior to study commencement indicating that most healthcare organisations were unlikely to be aligned with the evidence-based practice recommendations. While our analysis adjusted for baseline alignment, it is possible that a ceiling effect was reached in this group or other potentially confounding variables may have influenced differences in follow-up recommendation alignment. Formal assessment of evidence alignment and other potential confounders prior to study commencement was considered impractical within this study due to the data collection burden for time-limited hospital managers. Although, future research may benefit from these baseline assessments if information can be captured via routinely collected data for administrative or other purposes.
The decision to use an externally based, centralised knowledge broker to deliver support across multiple hospitals via interactive online webinar may have been an important factor influencing the strategy’s impact. Our findings align with a similar cluster randomised controlled trial conducted by Minian and colleagues, which compared generic emails with a remote knowledge broker to integrate mood management into a smoking cessation program [61]. In their study, the more intense and personalised remote knowledge broker strategy was no more effective at enabling healthcare professionals to provide their patients with mood management resources. The “more is better” theory, suggesting that a higher implementation strategy dosage through frequency of interactions or longer duration leads to greater success, is difficult to reconcile with this emerging empirical evidence. Hospital managers and healthcare professionals are time and resource constrained, facing multiple competing priorities that limit their ability to engage in more active strategies to facilitate evidence implementation [41]. The limited frequency and duration of the knowledge broker interactions may have impeded observed effectiveness of the strategy. However, it is unlikely that providing additional opportunities for engagement or a longer duration (e.g., 24 months) would have changed the dosage received, given the limited voluntary hospital manager engagement throughout the 12-month period and the desire to make an internal decision being the most common reason for limited engagement.
Most clusters only attended one or two group-based knowledge broker support sessions, which were delivered online rather than face-to-face. The limited dosage and mode of delivery may have constrained the ability to build relationships with the hospital managers. Knowledge broker roles are inherently relationship based [62], whose theory of change is premised on interpersonal contact [63], development of rapport [64], and building linkages and exchange between research producers and end users [31,39]. These social influence mechanisms of change [65–68] imply that the effectiveness of a knowledge broker may be largely individual dependent [53] and could be enhanced by embedded brokerage roles within organisations. For example, interactions between different hospital managers and knowledge brokers can result in varying levels of “relationship capital,” which is instrumental to fostering use of knowledge [63]. Less formal roles that are internal to organisations, such as opinion leaders or clinical champions, could instead be considered to leverage preexisting, peer-to-peer relationships and channels of persuasion [63]. It is important for informal brokers to consider group affiliation with the hospital managers, through occupation and professional legitimacy, to ensure that brokerage is not considered as “outsider expertise” [38]. These localised models of knowledge brokering, such as the National Institute for Health Research (NIHR) Collaborations for Leadership in Applied Health Research and Care (CLAHRCs), may provide a more situationally relevant, flexible, and collaborative approach to implement research into practice [69], albeit at a higher cost of investment.
The small number of clusters in this study was primarily driven by the need to cluster multiple geographically distinct hospitals to avoid contamination. During protocol design and sample size calculation, it was not possible to anticipate how many wards would be included in each cluster prior to recruitment, as we first needed to ascertain the decision-making structure for each healthcare organisation to in order to understand how many wards were nested within each decision-making structure (e.g., single hospital or network of hospitals). We acknowledge that our efforts to understand and cluster according to these decision-making structures prevented prespecification of cluster size. This design element was imperative to avoid the potential risk of contamination between units of randomisation. Recruitment challenges and difficulty ensuring adequate statistical power in analyses have been reported in other studies seeking to implement evidence into health system policy changes [70], particularly for pragmatic “real-world” projects at the level of organisations rather than individuals [71,72]. Considering the challenges experienced in this study and others, some have questioned the appropriateness of empirical designs for evaluating the effectiveness of implementation strategies, such as knowledge brokering [36]. Instead, more exploratory approaches oriented towards improving the understanding of how, why, when, and in what circumstances these strategies address more subjective, intermediate outcomes (e.g., capability for evidence engagement) have been advocated. These types of pluralistic, discursive, and iterative methods hold considerable value for understanding the processes by which the institutionalisation of evidence-based practice can occur. However, the decision to delegate and allocate healthcare resources to specific individual knowledge brokers requires consideration of its effectiveness and cost-effectiveness, in relation to competing priorities that are known to effectively improve health outcomes elsewhere in the health system. We contend that there are design-based solutions for overcoming the challenges in empirically studying the effectiveness of implementation strategies. For example, the use of counterbalanced implementation study designs has been proposed as a solution to these challenges, requiring smaller sample sizes through the concurrent investigation of multiple implementation strategies across different health context areas [73,74].
Weekend allied health service provision presented a uniquely challenging contextual area to implement evidence into practice. Healthcare professionals tend to incorporate local “tacit” or “codified” knowledge [75] into resource allocation decisions [76], particularly when presented with recommendations to reduce weekend allied health in acute general medical and surgical wards where the evidence base was considered less clear [58]. These findings align with previous research reporting that disinvestment from weekend services in these wards has been perceived as a threat to professional identity [77]. Conversely, recommendation to increase weekend services in subacute rehabilitation units was met with enthusiasm where this aligned with previously held attitudes, beliefs, and values [58]. Future research on these knowledge translation strategies is therefore needed across multiple contextual areas to determine the interaction effect between these strategies and context areas [73,74]. Further, an updated knowledge broker strategy with greater internal organisational relationships, which incorporates a component of cognitive debiasing to assist managers and policy makers when confronted with evidence that conflicts with current service delivery models, might reduce potential barriers to change and facilitate active implementation of evidence into practice [78–80].
Conclusions
Owing to a lack of power at the cluster level, this trial was unable to determine whether the use of a webinar-based knowledge broker was more effective than dissemination of recommendations or usual practice for promoting evidence-informed healthcare management decisions for inpatient weekend allied health services. The implication of this research is that more intense and interactive strategies for implementing evidence into practice do not always enable changes in resource allocation. Future research is needed across multiple contextual areas of healthcare recommendations to understand the context dependency of implementation strategy success, using counterbalanced implementation study deigns.
Supporting information
S1 Data. Cluster-level per day values (blinded).
https://doi.org/10.1371/journal.pmed.1003833.s001
(XLSX)
S2 Data. Ward-level per day values (blinded).
https://doi.org/10.1371/journal.pmed.1003833.s002
(XLSX)
S2 Text. CONSORT extension for cluster trials.
https://doi.org/10.1371/journal.pmed.1003833.s004
(DOCX)
S3 Text. Standards for Reporting Implementation Studies statement (StaRI).
https://doi.org/10.1371/journal.pmed.1003833.s005
(DOCX)
S4 Text. Weekend allied health recommendation.
https://doi.org/10.1371/journal.pmed.1003833.s006
(PDF)
Acknowledgments
The authorship team wish to acknowledge the contributions of Dr Jenni White and Dr Kellie Grant to the delivery of the implementation strategies and data collection within this study. We would also like to recognise support from the Victorian Department of Health and Human services.
References
- 1. Braithwaite J, Hibbert PD, Jaffe A, White L, Cowell CT, Harris MF, et al. Quality of Health Care for Children in Australia, 2012–2013. JAMA. 2018;319(11):1113–24. pmid:29558552
- 2. Runciman WB, Hunt TD, Hannaford NA, Hibbert PD, Westbrook JI, Coiera EW, et al. CareTrack: assessing the appropriateness of health care delivery in Australia. Med J Aust. 2012;197(2):100–5. pmid:22794056
- 3. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348(26):2635–45. pmid:12826639
- 4. Steel N, Bachmann M, Maisey S, Shekelle P, Breeze E, Marmot M, et al. Self reported receipt of care consistent with 32 quality indicators: national population survey of adults aged 50 or more in England. BMJ. 2008;337:a957. pmid:18703659
- 5. Saini V, Garcia-Armesto S, Klemperer D, Paris V, Elshaug AG, Brownlee S, et al. Drivers of poor medical care. Lancet. 2017;390(10090):178–90. pmid:28077235
- 6. Berwick DM, Hackbarth AD. Eliminating Waste in US Health Care. JAMA. 2012;307(14):1513–6. pmid:22419800
- 7. Braithwaite J, Glasziou P, Westbrook J. The three numbers you need to know about healthcare: the 60–30–10 Challenge. BMC Med. 2020;18(1):102. pmid:32362273
- 8. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581–629. pmid:15595944
- 9. Sheikh A. Decommissioning care: The need for rigorous multifaceted evaluations of decisions to withdraw health services. PLoS Med. 2017;14(10):e1002413. pmid:29088222
- 10. Campbell L, Bunston R, Colangelo S, Kim D, Nargi J, Hill K, et al. The Provision of Weekend Physiotherapy Services in Tertiary-Care Hospitals in Canada. Physiother Can. 2010;62(4):347–54. pmid:21886374
- 11. Shaw KD, Taylor NF, Brusco NK. Physiotherapy Services Provided Outside of Business Hours in Australian Hospitals: A National Survey. Physiother Res Int. 2013;18(2):115–23. pmid:23038214
- 12. Caruana EL, Kuys SS, Brauer SG. Allied health weekend service provision in Australian rehabilitation units. Australas J Ageing. 2018;37(2):E42–e8. pmid:29570236
- 13. Ottensmeyer CA, Chattha S, Jayawardena S, McBoyle K, Wrong C, Ellerton C, et al. Weekend physiotherapy practice in community hospitals in Canada. Physiother Can. 2012;64(2):178–87. pmid:23449882
- 14. Hill K, Brooks D. A description of weekend physiotherapy services in three tertiary hospitals in the greater Toronto area. Physiother Can 2010;62(2):155–62. pmid:21359048.
- 15. Haines TP, O’Brien L, Mitchell D, Bowles K-A, Haas R, Markham D, et al. Study protocol for two randomized controlled trials examining the effectiveness and safety of current weekend allied health services and a new stakeholder-driven model for acute medical/surgical patients versus no weekend allied health services. Trials. 2015;16(1):133. pmid:25873250
- 16. Haines TP, Bowles KA, Mitchell D, O’Brien L, Markham D, Plumb S, et al. Impact of disinvestment from weekend allied health services across acute medical and surgical wards: 2 stepped-wedge cluster randomised controlled trials. PLoS Med. 2017;14(10):e1002412. pmid:29088237
- 17. Sarkies MN, White J, Henderson K, Haas R, Bowles J. Additional weekend allied health services reduce length of stay in subacute rehabilitation wards but their effectiveness and cost-effectiveness are unclear in acute general medical and surgical hospital wards: a systematic review. J Physiother. 2018;64(3):142–58. pmid:29929739
- 18. Lane H, Sturgess T, Philip K, Markham D, Martin J, Walsh J, et al. What Factors Do Allied Health Take Into Account When Making Resource Allocation Decisions? Int J Health Policy Manag. 2018;7(5):412–20. pmid:29764105
- 19. Sarkies MN, Francis-Auton E, Long JC, Partington A, Pomare C, Nguyen HM, et al. Implementing large-system, value-based healthcare initiatives: a realist study protocol for seven natural experiments. BMJ Open. 2020;10(12):e044049. pmid:33371049
- 20. Campbell D, Donald B, Moore G, Frew D. Evidence check: knowledge brokering to commission research reviews for policy. Evid Policy. 2011;7(1):97–107.
- 21. Chambers D, Grant R, Warren E, Pearson S-A, Wilson P. Use of evidence from systematic reviews to inform commissioning decisions: a case study. Evid Policy. 2012;8(2):141–8.
- 22. Dobbins M, Cockerill R, Barnsley J, Ciliska D. Factors of the innovation, organization, environment, and individual that predict the influence five systematic reviews had on public health decisions. Int J Technol Assess Health Care. 2001;17(4):467. pmid:11758291
- 23. Grimshaw J, Eccles M, Thomas R, MacLennan G, Ramsay C, Fraser C, et al. Toward evidence-based quality improvement. Evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966–1998. J Gen Intern Med. 2006;21(Suppl 2):S14–20. pmid:16637955
- 24. Knapp JF, Simon SD, Sharma V. Does Active Dissemination of Evidence Result in Faster Knowledge Transfer Than Passive Diffusion?: An Analysis of Trends of the Management of Pediatric Asthma and Croup in US Emergency Departments From 1995 to 2009. Pediatr Emerg Care. 2015;31(3):190–6. pmid:24694945
- 25. Salbach NM, Wood-Dauphinee S, Desrosiers J, Eng JJ, Graham ID, Jaglal SB, et al. Facilitated interprofessional implementation of a physical rehabilitation guideline for stroke in inpatient settings: process evaluation of a cluster randomized trial. Implement Sci. 2017;12(1):100. pmid:28764752
- 26. Vedel I, Le Berre M, Sourial N, Arsenault-Lapierre G, Bergman H, Lapointe L. Shedding light on conditions for the successful passive dissemination of recommendations in primary care: a mixed methods study. Implement Sci. 2018;13(1):129. pmid:30326969
- 27. Lomas J. Diffusion, Dissemination, and Implementation: Who Should Do What? Ann N Y Acad Sci. 1993;703(1):226–37. pmid:8192299
- 28. Prior M, Guerin M, Grimmer-Somers K. The effectiveness of clinical guideline implementation strategies—a synthesis of systematic review findings. J Eval Clin Pract. 2008;14(5):888–97. pmid:19018923
- 29. Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. BMJ. 1998;317(7156):465–8. pmid:9703533
- 30. Lavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J. How Can Research Organizations More Effectively Transfer Research Knowledge to Decision Makers? Milbank Q. 2003;81(2):221–48. pmid:12841049
- 31. Lomas J. The in-between world of knowledge brokering. BMJ. 2007;334(7585):129–32. pmid:17235094
- 32. Conklin J, Lusk E, Harris M, Stolee P. Knowledge brokers in a knowledge network: the case of Seniors Health Research Transfer Network knowledge brokers. Implement Sci. 2013;8(1):7. pmid:23302517
- 33. Wahabi HA, Al-Ansary LA. Innovative teaching methods for capacity building in knowledge translation. BMC Med Educ. 2011;11(1):85. pmid:21999174
- 34. Gerrish K, McDonnell A, Nolan M, Guillaume L, Kirshbaum M, Tod A. The role of advanced practice nurses in knowledge brokering as a means of promoting evidence-based practice among clinical nurses. J Adv Nurs. 2011;67(9):2004–14. pmid:21507046
- 35. Ward V, Smith S, House A, Hamer S. Exploring knowledge exchange: A useful framework for practice and policy. Soc Sci Med. 2012;74(3):297–304. pmid:22014420
- 36. Dobbins M, Hanna SE, Ciliska D, Manske S, Cameron R, Mercer SL, et al. A randomized controlled trial evaluating the impact of knowledge translation and exchange strategies. Implement Sci. 2009;4(1):61. pmid:19775439
- 37. Cranley LA, Cummings GG, Profetto-McGrath J, Toth F, Estabrooks CA. Facilitation roles and characteristics associated with research use by healthcare professionals: a scoping review. BMJ Open. 2017;7(8):e014384. pmid:28801388
- 38. Currie G, White L. Inter-professional barriers and knowledge brokering in an organizational context: the case of healthcare. Organ Stud. 2012;33(10):1333–61.
- 39. Ward V, House A, Hamer S. Knowledge Brokering: The missing link in the evidence to action chain? Evid Policy. 2009;5(3):267–79. pmid:21258626
- 40. Chew S, Armstrong N, Martin G. Institutionalising knowledge brokering as a sustainable knowledge translation solution in healthcare: how can it work in practice? Eviden Policy. 2013;9(3):335–51.
- 41. Sarkies MN, Bowles KA, Skinner EH, Haas R, Lane H, Haines TP. The effectiveness of research implementation strategies for promoting evidence-informed policy and management decisions in healthcare: a systematic review. Implement Sci. 2017;12(1):132. pmid:29137659
- 42. Kislov R, Wilson P, Boaden R. The ‘dark side’ of knowledge brokering. J Health Serv Res Policy. 2017;22(2):107–12.
- 43. Hurtubise K, Rivard L, Héguy L, Berbari J, Camden C. Virtual Knowledge Brokering: Describing the Roles and Strategies Used by Knowledge Brokers in a Pediatric Physiotherapy Virtual Community of Practice. J Contin Educ Heal Prof. 2016;36(3):186–94. pmid:27583995
- 44. Sarkies MN, White J, Morris ME, Taylor NF, Williams C, O’Brien L, et al. Implementation of evidence-based weekend service recommendations for allied health managers: a cluster randomised controlled trial protocol. Implement Sci. 2018;13(1):60. pmid:29690882
- 45. Haas R, Sarkies M, Bowles KA, O’Brien L, Haines T. Early commencement of physical therapy in the acute phase following elective lower limb arthroplasty produces favorable outcomes: a systematic review and meta-analysis examining allied health service models. Osteoarthr Cartil. 2016;24(10):1667–81. pmid:27224276
- 46. Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687. pmid:24609605
- 47. Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8(1):139. pmid:24289295
- 48. Davies P. The State of Evidence-Based Policy Evaluation and its Role in Policy Formation. Natl Inst Econ Rev. 2012;219:R41–52.
- 49. Michie S, van Stralen MM, West R. The behaviour change wheel: A new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6(1):42. pmid:21513547
- 50. Slade SC, Philip K, Morris ME. Frameworks for embedding a research culture in allied health practice: a rapid review. Health Res Policy Syst. 2018;16(1):29. pmid:29562939
- 51. Uneke CJ, Sombie I, Johnson E, Uneke BI, Okolo S. Promoting the use of evidence in health policymaking in the ECOWAS region: the development and contextualization of an evidence-based policymaking guidance. Glob Health. 2020;16(1):73. pmid:32762759
- 52. Grimshaw J, Thomas R, MacLennan G, Fraser C, Ramsay C, Vale L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004;8(6):1–72. pmid:14960256
- 53. Dagenais C, Laurendeau M-C, Briand-Lamarche M. Knowledge brokering in public health: a critical analysis of the results of a qualitative evaluation. Eval Program Plann. 2015;53:10–7. pmid:26203522
- 54. Sarkies MN, Bowles KA, Skinner EH, Mitchell D, Haas R, Ho M, et al. Data collection methods in health services research: hospital length of stay and discharge destination. Appl Clin Inform. 2015;6(1):96–109. pmid:25848416
- 55.
Sealed Envelope Ltd. 2018. Create a blocked randomisation list [Online]. [cited 14 Sept 2021]. https://www.sealedenvelope.com/simple-randomiser/v1/lists.
- 56.
Higgins JP, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane handbook for systematic reviews of interventions. Chichester (UK): John Wiley & Sons; 2019.
- 57.
SankeyMATIC diagram builder. [cited 14 Sept 2021]. http://sankeymatic.com/.
- 58. White JGK, Sarkies M, Evidence Translation in Allied Health (EviTAH) Group, Haines T. Translating evidence into practice: A longitudinal qualitative exploration of allied health decision making. Health Res Policy Syst. 2021;19:38.
- 59. Dobbins M, Traynor RL, Workentine S, Yousefi-Nooraie R, Yost J. Impact of an organization-wide knowledge translation strategy to support evidence-informed public health decision making. BMC Public Health. 2018;18(1):1412. pmid:30594155
- 60. Sarkies M, Long JC, Pomare C, Wu W, Clay-Williams R, Nguyen HM, et al. Avoiding unnecessary hospitalisation for patients with chronic conditions: a systematic review of implementation determinants for hospital avoidance programmes. Implement Sci. 2020;15(1):91. pmid:33087147
- 61. Minian N, Ahad S, Ivanova A, Veldhuizen S, Zawertailo L, Ravindran A, et al. The effectiveness of generic emails versus a remote knowledge broker to integrate mood management into a smoking cessation programme in team-based primary care: a cluster randomised trial. Implement Sci. 2021;16(1):30. pmid:33743777
- 62. Bornbaum CC, Kornas K, Peirson L, Rosella LC. Exploring the function and effectiveness of knowledge brokers as facilitators of knowledge translation in health-related settings: a systematic review and thematic analysis. Implement Sci. 2015;10(1):162. pmid:26589972
- 63. Thompson GN, Estabrooks CA, Degner LF. Clarifying the concepts in knowledge transfer: a literature review. J Adv Nurs. 2006;53(6):691–701. pmid:16553677
- 64. Traynor R, DeCorby K, Dobbins M. Knowledge brokering in public health: a tale of two studies. Public Health. 2014;128(6):533–44. pmid:24684852
- 65. Carpenter D, Esterling K, Lazer D. The strength of strong ties: A model of contact-making in policy networks with evidence from US health politics. Ration Soc. 2003;15(4):411–40.
- 66. Contandriopoulos D, Lemire M, Denis JL, Tremblay É. Knowledge exchange processes in organizations and policy arenas: a narrative systematic review of the literature. Milbank Q. 2010;88(4):444–83. pmid:21166865
- 67. Nutley S, Walter I, Davies HT. Promoting evidence-based practice: Models and mechanisms from cross-sector review. Res Soc Work Pract. 2009;19(5):552–9.
- 68. Wilson KM, Brady TJ, Lesesne C, NCCDPHP Work Group on Translation. An organizing framework for translation in public health: the knowledge to action framework. Prev Chronic Dis. 2011;8(2). pmid:21324260
- 69. Harvey G, Fitzgerald L, Fielden S, McBride A, Waterman H, Bamford D, et al. The NIHR collaboration for leadership in applied health research and care (CLAHRC) for Greater Manchester: combining empirical, theoretical and experiential evidence to design and evaluate a large-scale implementation strategy. Implement Sci. 2011;6(1):96. pmid:21861886
- 70. Foy R, Sales A, Wensing M, Aarons GA, Flottorp S, Kent B, et al. Implementation science: a reappraisal of our journal mission and scope. Implement Sci. 2015;10(1):51. pmid:25928695
- 71. Johnson AM, Jones SB, Duncan PW, Bushnell CD, Coleman SW, Mettam LH, et al. Hospital recruitment for a pragmatic cluster-randomized clinical trial: Lessons learned from the COMPASS study. Trials. 2018;19(1):74. pmid:29373987
- 72. McDonald AM, Knight RC, Campbell MK, Entwistle VA, Grant AM, Cook JA, et al. What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials. 2006;7(1):9. pmid:16603070
- 73. Sarkies MN, Skinner EH, Bowles K-A, Morris ME, Williams C, O’Brien L, et al. A novel counterbalanced implementation study design: methodological description and application to implementation research. Implement Sci. 2019;14(1):45. pmid:31046788
- 74. Sarkies MN, Maloney S, Symmons M, Haines TP. Video strategies improved health professional knowledge across different contexts: a helix counterbalanced randomized controlled study. J Clin Epidemiol. 2019;112:1–11. pmid:30998965
- 75.
Kislov R, Wilson P, Cummings G, Ehrenberg A, Gifford W, Kelly J, et al. The Chain of Codified Knowledge: Organisational Enactment of Evidence-Based Health Care in Four High-Income Countries. Transitions and Boundaries in the Coordination and Reform of Health Services. Switzerland AG: Springer. 2020:243–67.
- 76. Wenzel L-A, White J, Sarkies MN, Morris ME, Carey L, Williams C, et al. How do health professionals prioritize clinical areas for implementation of evidence into practice? A cross-sectional qualitative study. JBI Evid Implement. 2020;18(3):288–96. pmid:32516206
- 77. Mitchell D, O’Brien L, Bardoel A, Haines T. Understanding Health Professional Responses to Service Disinvestment: A Qualitative Study. Int J Health Policy Manag. 2019;8(7):403–11. pmid:31441277
- 78. Morewedge CK, Yoon H, Scopelliti I, Symborski CW, Korris JH, Kassam KS. Debiasing Decisions:Improved Decision Making With a Single Training Intervention. Policy Insights Behav Brain Sci. 2015;2(1):129–40.
- 79. Croskerry P, Singhal G, Mamede S. Cognitive debiasing 1: origins of bias and theory of debiasing. BMJ Qual Saf. 2013;22(Suppl 2):ii58–64. pmid:23882089
- 80. Croskerry P, Singhal G, Mamede S. Cognitive debiasing 2: impediments to and strategies for change. BMJ Qual Saf. 2013;22(Suppl 2):ii65–72. pmid:23996094