Participants

National Epidemiologic Survey on Alcohol and Related Conditions study.

The National Epidemiologic Survey on Alcohol and Related Conditions (NESARC) study [25,44,45] is a large, longitudinal general population study conducted by the National Institute on Alcohol Abuse and Alcoholism in the US. The initial NESARC study included a nationally representative sample of 43,093 participants assessed for a wide array of psychiatric and substance use issues in 2001 to 2002 [25]. Wave 2 occurred 3 years later and included follow-up interviews with 34,653 of the participants from Wave 1 (see Grant and colleagues [25,44,45] for additional details regarding study procedures for the NESARC project). The current analyses were restricted to the 34,641 NESARC participants who participated in Waves 1 and 2 and had follow-up suicide attempt data available from Wave 2. The random selection procedure from the IBM SPSS Statistics 24 software package was used to generate 4 random subsets of participants from the NESARC dataset, including 2 for development [NESARC 1 (N = 8,872) and NESARC 2 (N = 8,525)] and 2 for validation [NESARC 3 (N = 8,516) and NESARC 4 (N = 8,728), see Table 1 for sample characteristics]. Sampling was performed without replacement to ensure that each case was not selected more than once. Note that the 4 subsets of participants from NESARC did not differ by rate of prospective suicide attempts, p = 0.973; lifetime suicide attempts, p = 0.729; gender, p = 0.541; age, p = 0.448; race, p = 0.814; sexual orientation, p = 0.839; education, p = 0.343; income, p = 0.67; or employment status, p = 0.923. See Grant and colleagues [25,44,45] for additional details regarding study procedures for the NESARC study.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Descriptive statistics for the longitudinal samples used to develop and validate the DRS.

https://doi.org/10.1371/journal.pmed.1003713.t001

Main outcome variable

The present analyses focused on the prediction of future suicide attempts (as opposed to death by suicide) for several reasons. First, death by suicide is an extremely rare event. In the US, the age-adjusted rate of suicide was 13.9/100,000 in 2019 [2]. Suicide attempts are far more common than suicide deaths [2,31,32] and are among the strongest known predictors of death by suicide [3,31,32]. Indeed, Olfson and colleagues [32] found that 1.6% of individuals who attempted suicide died by suicide within 12 months, whereas 3.9% died by suicide within 5 years. Suicide attempts are also routinely assessed in high-quality longitudinal datasets, whereas there are few, if any, longitudinal research databases with sufficiently large samples sizes to study death by suicide that also contain high-quality, systematically assessed data on established predictors of suicidal behavior collected via rigorous research-based assessments. Of note, Belsher and colleagues [24] recently recommended that future suicide risk models target more common outcomes, including suicide attempts specifically, to develop better performing models of suicide risk following their review of existing suicide risk models. It is also important to recognize that suicide attempts are highly serious events in their own right. As noted by the World Health Organization [48], “Suicide attempts result in a significant social and economic burden for communities due to the utilization of health services to treat the injury, the psychological and social impact of the behaviour on the individual and his/her associates and, occasionally, the long-term disability due to the injury.”

Overview of the analysis plan

The primary analyses underlying the development and validation of the DRS began in April 2018 and ended in July 2020 and were conducted under research protocols approved by the Institutional Review Boards of the Durham Veterans Affairs Health Care System, Duke University School of Medicine, and the VA Boston Healthcare System. Additional analyses requested by reviewers during the peer review process were conducted from March 2021 to April 2021. While a written prospective analysis plan was not developed prior to initiating work on this project, a systematic approach was used to develop and validate the DRS. Specifically, measure development began with a review of the extant literature on risk factors for suicidal behavior [3–18,21–24,26,27,29–43]. After identifying and ranking a wide array of potential longitudinal predictors of death by suicide and suicide attempts from the literature, secondary data analyses were conducted to develop the DRS in the development samples (i.e., NESARC 1, NESARC 2, and REHAB; combined N = 17,630). It was then tested in the validation samples (i.e., NESARC 3, NESARC 4, and VALOR; combined N = 18,024) to determine if it continued to be predictive in separate cohorts of similar size and composition. This study is reported as per the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) reporting guideline (see S1 TRIPOD Checklist).

Development of the Durham Risk Score

Because our primary goal was to develop a suicide attempt risk checklist that could be used by clinicians to reliably discriminate high-risk patients from low-risk patients, receiver operating characteristic (ROC) curve analysis was the primary statistical approach used to develop the DRS in the development samples. Logistic regression was also utilized to help guide variable selection procedures in some instances. ROC curves, correlation matrices, and chi-squared tests were used to evaluate bivariate associations and to identify optimal cut points or iterations of variables that were maximally predictive of suicide attempts.

We elected to split the NESARC sample into 4 smaller samples to ensure that we would have (1) 2 large datasets with which to conduct the initial development work in; and (2) 2 large datasets of similar size and composition in which to test the performance of the final selected model. That is, consistent with standard holdout cross-validation approaches that utilize a training dataset (T_tr) and a validation dataset (T_v) to avoid overfitting due to limiting the development sample to a single dataset, we utilized 2 large, randomly selected subsets of NESARC participants to develop the DRS. A third sample (REHAB), which was smaller, collected independently, and comprised entirely of veterans (many of whom had psychiatric disorders), was also included in the development phase to further protect against overfitting and to increase generalizability of findings. Thus, from a total sample size of 35,654 participants, 17,630 participants (including NESARC 1, NESARC 2, and REHAB) were utilized to develop the DRS, whereas the remaining samples (NESARC 3, NESARC 4, and VALOR; combined N = 18,024) were held out to test the performance of the DRS in testing datasets (Tt) of similar size and composition. Table 1 provides descriptive statistics for each of the samples included in the present analyses.

Consistent with recommendations for building appropriate and stable predictive models [50,51], independent variable selection was guided by theory [29,30], prior empirical investigations [3–5,7–18,21–24,26,27,29–43], clinical considerations [3–18,21–24,29–31,48], univariate and bivariate statistical analyses, and consideration of multicollinearity among independent variables. Accordingly, independent variable selection and screening began with a review of the relevant literature concerning risk factors for suicidal behavior [3–5,7–18,21–24,26,27,29–43]. An a priori decision was made to prioritize variables that had particularly strong empirical support as longitudinal risk factors in the literature (e.g., recent psychiatric hospitalization)—even if their effects were less pronounced in our specific samples—in hopes of increasing the stability and replicability of the checklist in future work.

To simplify quantification of the empirical evidence, we relied on Franklin and colleagues’ [3] meta-analysis, which, in our opinion, was the most comprehensive work on this subject available at the time of the analyses. The top 10 broad risk categories for suicide deaths and suicide attempts were assigned scores from 1 to 10, where a score of 10 was assigned to the broad risk categories most strongly associated with suicide deaths and attempts. In addition, the top 5 predictors of suicide deaths and suicide attempts identified in this meta-analysis were also assigned scores from 6 to 10. Thus, potential evidence scores ranged from 0 to 40 (see Table A in S1 File). Table 2 provides the empirical evidence score that we assigned to each of the variables based on the findings from Franklin and colleagues [3] as well as the potential impact of each variable’s entry into the model on the cumulative AUC value for different iterations of the DRS across the 3 development samples.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Empirical evidence scores and impact on cumulative AUC values for each of the DRS variables across the 3 development samples.

https://doi.org/10.1371/journal.pmed.1003713.t002

As can be seen in Table 2 and Table A in S1 File, a history of prior suicide attempts was the variable with the highest total empirical evidence score based on this approach (total empirical evidence score = 35; mean AUC = 0.62), whereas psychosis/schizophrenia was the lowest scoring variable (total empirical evidence score = 1; mean AUC = 0.52) that was considered. As can be seen in Fig A in S1 File, a statistically significant positive correlation (r = 0.37, p < 0.001) was observed between the total empirical evidence score and the mean bivariate AUC value for each construct considered across the 3 development samples, providing support for our general (albeit simplistic) approach to quantifying the empirical evidence for the variables considered.

To ensure that scoring and interpretation remained as simple as possible (i.e., to ensure that higher scores would equal higher risk), an a priori decision was also made to only include dichotomous risk factors with obvious main effects. Thus, protective factors, risk factors that only had effects in the presence of other variables (e.g., through interactions), and scaled risk factors were excluded as potential predictors (although in several cases we were able to successfully dichotomize items collected on a scale, e.g., sleep problems and perceived health). Additionally, consistent with Babyak’s recommendations [50], overlapping constructs were aggregated in many instances to increase model stability and reduce the number of variables included in the checklist. As a result, composite variables were created for “mood disorders,” “substance use disorders,” “violence/incarceration,” “sexual abuse/sexual assault,” and “lesbian, gay, bisexual, transgender, and queer or questioning (LGBTQ)”.

An iterative, sequential approach to model building was taken whereby variables expected to have strong and pronounced effects on future risk for suicide based on the extant literature (e.g., prior suicide attempts, hospitalization, NSSI, and suicidal ideation) [3] were entered before variables with less empirical support (e.g., demographic predictors). We began by calculating ROC curves for each of the potential predictors across the 3 development samples (see Table A in S1 File). Then, beginning with the 2 variables we identified as having the strongest empirical support from the literature (i.e., prior suicide attempts and prior psychiatric hospitalization), we evaluated if the combination (i.e., sum) of these 2 variables resulted in an AUC value that was consistently higher across the development samples than the AUC values for the individual variables when examined separately. Utilizing this general approach, we systematically evaluated each new variable for potential inclusion in the checklist until we were unable to identify any additional variables that improved discrimination of high-risk individuals from low-risk individuals in 1 or more of the development samples (see Table 2).

The final set of constructs selected for inclusion in the DRS are provided in Table 2, which also shows the impact of each variable’s entry into the model on the cumulative AUC value for different iterations of the DRS across the 3 development samples. It is, however, important to note that an iterative approach was taken to variable selection and that the constructs ultimately selected for inclusion in the DRS were those constructs that not only optimized predictive validity across the 3 development samples, but were also logical from both a theoretical and clinical perspective [18,29,30]. Other variables from the extant literature [3,18,21–23,29–43] were also considered (see Table A in S1 File), but not ultimately selected, including (among others) other psychiatric disorders (e.g., schizophrenia and anxiety disorders), recent life stressors, and various demographic variables (e.g., marital status). Different orders and iterations of variables (e.g., frequency, severity, and time frame of assessment) were also considered in order to optimize the predictive value of variables within the development samples. Please also note that many other potentially important variables (e.g., suicidal intent, access to lethal means, suicide plans, and a psychiatric hospitalization during the past 30 days) were not available for analysis in the samples utilized in the present study.

To be retained in the final version of the checklist, each variable needed to (1) have clear empirical support in the literature; (2) demonstrate a positive bivariate association with future suicide attempts in 1 or more of the development samples; (3) evidence incremental validity in 1 or more of the development samples; and (4) show minimal negative impact on incremental validity in the remaining development samples. Utilizing the approach described above, we initially selected 23 items for inclusion in the checklist, each weighted equally. Once we reached the point at which we were no longer able to identify any new variables that further improved the predictive utility of the score, we examined if doubling the weight by adding an additional point to the sum score of any of the items identified as top predictors further improved AUC values. This analysis revealed that doubling the weight of 4 of the top longitudinal predictors identified by Franklin and colleagues [3] (i.e., lifetime history of suicide attempts, psychiatric hospitalization, NSSI, and borderline personality disorder [BPD]) further improved the overall AUC value in 1 or more of the development samples (see Table 2).

Evaluation of the Durham Risk Score

ROC curves and logistic regression analyses were used to evaluate the discriminative ability of the DRS across the samples. Signal detection analysis was used to identify an optimal cut score [52] and to develop risk groups to facilitate interpretation of scores. Concentration of risk was evaluated [23], and rates of attempts, risk ratios, ORs, and 95% CIs were calculated for risk groups. ROC curves were also calculated in subgroups of interest, including women, men, Black, White, Hispanic, lower-income individuals, younger adults, veterans, LGBTQ individuals, as well as individuals with and without a history of suicidal thoughts and behaviors.

Missing data

Although we are strong proponents of multiple imputation and maximum likelihood estimation methods to handle missingness in most situations, we elected to treat missing data as absent (i.e., “0”) in the calculation of DRS scores in the present research because (1) this approach best reflects real-world clinical practice; and (2) some variables were systematically missing across different studies because they were not assessed as part of the study protocol. The only exception to this approach was for the VALOR sample, which was used to validate the DRS. Specifically, because the VALOR study protocol only assessed 15 of the 23 (i.e., 65%) variables used to calculate the DRS, VALOR analyses were limited to participating veterans (N = 780) with follow-up attempt data as well as complete data for all 15 of these variables to ensure that participants in the VALOR analyses had no more than 35% missing data.

Measures

Table B in S1 File summarizes the items and measures used to assess each of the 23 constructs included in the DRS across studies. Measures used to index the various constructs included well-validated structured interviews, such as the C-SSRS [5], SITBI [49], the Structured Clinical Interview for DSM (SCID) [53], the Clinician-Administered PTSD Scale (CAPS) [54], the Mini International Neuropsychiatric Interview (MINI)] [55], and the Alcohol Use Disorder and Associated Disabilities Interview Schedule (AUDADIS) [56], as well as a variety of self-report instruments, including the Alcohol Use Disorders Identification Test (AUDIT) [57], Drug Abuse Screening Test (DAST) [58], Davidson Trauma Scale (DTS) [59], Patient Health Questionnaire-9 (PHQ-9) [60], Traumatic Life Events Questionnaire (TLEQ) [61], Childhood Trauma Questionnaire (CTQ) [62], Veterans Rand 12-Item Health Survey (VR-12) [63], the Life Events Checklist (LEC) [64], and the Symptom Checklist-90 (SCL-90) [65]. Study-specific questionnaires [25–27,44–47] were also used to assess constructs in some instances, particularly those related to demographic characteristics.

As can be seen in Table B in S1 File, in the vast majority of cases, all of the specific items used to assess the constructs included in the calculation of the DRS score were assessed at the time of the baseline assessment; however, there were 8 instances in which at least a part of a construct of interest was only assessed at the time of the Wave 2 assessment. Such items are clearly marked in bold in Table B in S1 File. In each case in which an item from a follow-up wave was included in the assessment of 1 of the 23 items included in the calculation of the DRS score, we carefully considered both the nature of the specific item as well as the nature of the construct in general before making an a priori decision about whether to use a specific variable from a specific sample in the calculation of the DRS. In each case where such a variable was included in the assessment of a given construct, we felt that inclusion of the data from a given item was justified, given that our overall goal was to make the best possible suicide attempt risk checklist in order to enhance clinical care.

Most of these instances occurred in the NESARC dataset. For example, childhood sexual abuse, childhood physical abuse, and being jailed or sent to a juvenile detention center prior to the age of 18 were not assessed at the time of the NESARC baseline assessment. We reasoned that, given that reporting of these items during adulthood would have still involved retrospective reporting, even if they had been administered during the baseline assessment, it was reasonable (though not ideal) to include information regarding these important childhood experiences from Wave 2 in calculating the DRS score. Information from the Wave 2 NESARC interview was also used to index PTSD, BPD, and NSSI. In the case of PTSD, interviewers were required to retrospectively establish if symptoms of PTSD had begun prior to the baseline interview and whether they had been present from the time of the baseline interview to the time of the time of the Wave 2 interview. In the case of BPD and NSSI (which was assessed as part of the BPD interview), because BPD is a personality disorder that should be present by early adulthood, NESARC interviewers were instructed to frequently precede BPD questions (including NSSI) with “Most of the time throughout your life, regardless of the situation or whom you were with…”. Given the way that these questions were asked, and the fact that BPD and NSSI are among the strongest predictors of suicide attempts [3,35], we felt that it was critical to include these items in the calculation of the DRS score. Sexual orientation was also not assessed at the time of the NESARC baseline assessment. Although we recognize that sexual orientation can change over time, given the importance of systematically assessing this construct in relation to risk for suicidal behavior [36], we felt that it was also important to include the sexual orientation variable collected at NESARC Wave 2 in the calculation of the DRS score.

We fully recognize the problems associated with including a subset of variables assessed cross-sectionally in a checklist designed to prospectively predict suicide attempts in clinical settings and would have strongly preferred to have only included variables assessed longitudinally in the present analyses; however, such an approach would have precluded us from including several of the most well-established longitudinal predictors of suicide attempts (e.g., NSSI) [3]. In recognition of this challenging situation, we conducted a sensitivity analysis in the NESARC dataset to evaluate the performance of the DRS when these variables were excluded from the calculation of the score. As subsequently described in the Results section below, we were pleased to find that this sensitivity analysis revealed that the DRS continued to perform quite well (AUC = 0.86) in the NESARC validation cohort when all variables collected at Wave 2 were excluded from the calculation of the score, indicating that the core DRS measure is, in fact, a robust prospective predictor of suicide attempts, regardless of whether the items assessed cross-sectionally are included or not.

Lifetime history of NSSI was also assessed cross-sectionally in the VALOR sample. Again, however, the interviewers for this study were required to determine if the participant’s history of NSSI was present prior to the baseline interview, and only individuals who were retrospectively determined to have engaged in NSSI prior to the baseline interview were coded as having a history of lifetime NSSI at the time of the baseline interview. Given the manner in which NSSI was assessed in VALOR, as well as the importance of NSSI to suicide attempt risk prediction [3], we felt that inclusion of this data was also justified, given our primary goal of developing the best possible clinical assessment to facilitate identification of at-risk individuals.

Descriptive statistics and distribution of scores

Descriptive statistics for the DRS (Fig 1) across the different samples are provided in Table 1. As would be expected for a suicide attempt risk score, the DRS was positively skewed (1.7) and kurtotic (4.7) in the overall sample (Fig B in S1 File); however, among the 288 individuals who made a prospective suicide attempt during the follow-up period, DRS scores were normally distributed (M = 9.9; SD = 4.4; skewness = 0.33; kurtosis = −0.7; range: 1 to 22; Fig C in S1 File).

Logistic regression analyses

Logistic regression analyses were conducted to examine the predictive utility of the continuous DRS score as a predictor of prospective suicide attempts across the samples (Table 3). These analyses revealed a consistent pattern of increasing risk as a function of DRS score in both the combined development (OR = 1.48, 1.43 to 1.53, p < 0.001; Nagelkerke pseudo R² = 0.27) and validation samples (OR = 1.51, 1.46 to 1.56, p < 0.001; Nagelkerke pseudo R² = 0.29). Thus, for each additional point increase on the DRS, the odds of making a prospective suicide attempt increased by approximately 50% in both the combined development and validation cohorts.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Summary of logistic regression findings.

https://doi.org/10.1371/journal.pmed.1003713.t003

Receiver operating characteristic curve analyses

ROC analyses revealed that the overall AUC for the DRS total score in the combined development sample (total N = 17,630; Table 4) was 0.91 (0.89 to 0.93; Fig 2A). More importantly, the DRS continued to demonstrate excellent discriminative ability in the 3 validation samples excluded from the development analyses (combined validation sample AUC = 0.92, 0.90 to 0.94, N = 18,024; Table 4, Fig 2A), suggesting that our approach to instrument development was successful in protecting against overfitting [50].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. ROC curves for the DRS across subsets of participants.

Fig 2a: ROC curves for the DRS in the combined development and validation cohorts. Black line = development cohort; red line = validation cohort. Fig 2b: ROC curves for the risk groups in the development and validation cohorts. Black line = development cohort; red line = validation cohort. Fig 2c: ROC curves for the DRS among male and female participants. Black line = men; red line = women. Fig 2d: ROC curves for the DRS among White, Black, and Hispanic participants. Black line = White participants; red line = Black participants; blue line = Hispanic participants. Fig 2e: ROC curves for DRS among veteran, lower-income, younger, and LGBTQ participants. Black line = veteran participants; red line = lower-income participants; blue line = participants under the age of 35; orange line = LGBTQ participants. AUC, area under the curve; DRS, Durham Risk Score; LGBTQ, lesbian, gay, bisexual, transgender, and queer or questioning; ROC, receiver operating characteristic.

https://doi.org/10.1371/journal.pmed.1003713.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Areas under the curve for DRS and risk group status across samples and subsets of participants.

https://doi.org/10.1371/journal.pmed.1003713.t004

Subgroup analyses

Subgroup-based ROC analyses (Table 4, Fig 2C–2E) revealed that the DRS performed well among women (AUC = 0.91, 0.89 to 0.93), men (AUC = 0.93, 0.90 to 0.95), Black (AUC = 0.92, 0.88 to 0.96), White (AUC = 0.93, 0.91 to 0.95), Hispanic (AUC = 0.89, 0.86 to 0.93), veterans (AUC = 0.91, 0.89 to 0.94), lower-income individuals (AUC = 0.90, 0.88 to 0.92), younger adults (AUC = 0.88, 0.85 to 0.91), and LGBTQ individuals (AUC = 0.88, 0.81 to 0.94). In addition, as expected, participants with a history of suicidal thoughts or behaviors (N = 3,489) were significantly more likely to make a prospective suicide attempt (4.8% versus 0.4%; OR = 13.3, 10.5 to 16.9, p < 0.001) than those without a history of suicidal thoughts or behaviors at baseline; however, even within this high-risk subgroup, the DRS demonstrated good utility (AUC = 0.82, 0.79 to 0.85; Fig 2F). We further observed that 42% of prospective suicide attempts occurred among individuals who reported no lifetime history of suicidal thoughts or behavior at the time of the baseline assessment. Notably, the DRS (AUC = 0.88, 0.85 to 0.91; Fig 2F) also performed well in this important, but understudied subgroup.

Concentration of risk

A concentration of risk analysis [23] found that across all 35,654 participants, 82% of observed prospective suicide attempts occurred among individuals in the top 15% of DRS scores; 58% occurred in the top 5%; and 27% occurred in the top 1% (Fig 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Concentration of risk of prospective suicide attempts across all participants (N = 35,654).

DRS, Durham Risk Score.

https://doi.org/10.1371/journal.pmed.1003713.g003

Signal detection analyses

Signal detection analysis was used to identify a cut score that simultaneously maximized sensitivity and specificity and would be appropriate in typical clinical screening situations (Table 5). For situations in which a single clinical cut score is needed to identify at-risk individuals, we recommend a cut score of 6 or greater (corresponding to moderate-risk group status or approximately top 15% of scores), as this score had the highest overall J statistic [52] (i.e., Youden index = 0.68) and maximized both sensitivity (82%) and specificity (86%). A cut score of 6 also produced a PPV of 4% and a negative predictive value (NPV) of 100%. In contrast, for situations in which higher levels of PPV are preferred, a cut score of 9 or higher (corresponding to high-risk group status or approximately top 5% of scores) resulted in specificity of 96%, sensitivity of 58%, PPV of 10%, and NPV of 100%, whereas a cut score of 15 or higher (corresponding to highest-risk group status or approximately top 0.5% of scores) resulted in specificity of 100%, sensitivity of 18%, PPV of 27%, and NPV of 99% (Table 5).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Summary of the signal detection analyses for different DRS cutoffs in the total sample (N = 35,654).

https://doi.org/10.1371/journal.pmed.1003713.t005

Suicide attempt risk groups

To facilitate rapid interpretation of scores, suicide attempt risk groups were identified that appeared to correspond to clinically meaningful increases in suicide attempt risk based on the signal detection analyses described above. Table C in S1 File provides rates of attempts, odds, and predicted probabilities by risk group status for the total sample (N = 35,654). The AUC for risk group status was 0.90 (0.87 to 0.92) in the development sample and 0.91 (0.88 to 0.93) in the validation sample (Table 4, Fig 2B), indicating that our 6-group classification system was nearly as accurate at predicting future attempts as the DRS total score. Moreover, as expected, each increasing group level was associated with a marked increase in risk (Table 3). For example, the odds of making a prospective suicide attempt were more than 1,800 times greater in the highest-risk group (top 0.6% of scores; suicide attempt rate = 32.0%) relative to the lowest-risk group (bottom 43.5% of scores; suicide attempt rate = 0.03%) in the validation cohort (OR = 1,845.6, 433.6 to 7,855.3, p < 0.001; Table 3). Notably, as was the case for the total DRS score, risk group status appeared to confer similar increases in risk to participants regardless of whether or not they reported a history of suicidal thoughts or behavior at the time of the baseline assessment (Table 4, Fig 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Rates of attempts by risk group status among participants with and without a history of suicidal thoughts and behaviors (N = 35,654).

https://doi.org/10.1371/journal.pmed.1003713.g004

Sensitivity analysis to examine the impact of variables assessed at Wave 2

To assess the potential impact of the variables that were assessed at Wave 2 in the NESARC study on the performance of the DRS, a sensitivity analysis was conducted in which the 7 items from Wave 2 (i.e., sexual abuse/assault, child physical abuse, history of juvenile detention, NSSI, BPD, PTSD, and LGBTQ status) were excluded from the computation of the DRS score. Please note that item #W2S11Q6A (which was also assessed at Wave 2) was also removed from the calculation of lifetime history of violence and incarceration for these analyses. Thus, all variables used in the calculation of the DRS in these sensivity analyses were assessed at the time of the Wave 1 interview and prior to the occurence of any prospective suicide attempts that occurred between Waves 1 and 2. This analysis (Table D in S1 File) revealed that the abbreviated version of the DRS based entirely on variables collected during the baseline assessment continued to perform quite well (AUC = 0.86) in the combined NESARC validation cohort (i.e., NESARC 3 and NESARC 4, combined N = 17,244).

Association between number of items assessed and AUC values

A robust positive correlation was observed between number of items assessed and AUC values across the 6 samples (r = 0.94, p = 0.006; see Fig D1 in S1 File). Accordingly, we recommend that, whenever possible, clinicians and researchers who wish to use the DRS systematically assess and score all 23 items using the most reliable and valid assessment methods available to them at the time of the assessment (see S1 Durham Risk Score Guide for additional details). We also assessed the association between number of items and AUC values with the 4 additional sensitivity samples included (Fig D2 in S1 File) and observed that there continued to be a robust positive association between number of items assessed and AUC values (r = 0.83, p = 0.003). Moreover, as can be seen in Table D and Fig D2 in S1 File, 3 of the 4 NESARC sensitivity samples (AUC range: 0.85 to 0.87) had AUC values that were higher than the AUC for the VALOR sample (0.82), which also only assessed 15 items. Thus, the lower AUC values observed in the NESARC sensitivity analyses (i.e., when only 15 items were included in the calculation of the total score) were highly consistent with the pattern of findings one would expect based on the positive association observed between items assessed and AUC values.

Prediction of suicide attempts occurring outside of the context of mood disorders

During the peer review process, reviewers noted that a potential limitation of our choice to utilize the question from the Low Mood portion of the NESARC interview to define prospective suicide attempts (i.e., “During that time since your LAST interview when (your mood was at its lowest/you enjoyed or cared least about things), did you attempt suicide?”) was that our findings might be not be generalizable to the prediction of suicide attempts occurring outside of the context of mood disorders (e.g., during psychosis). To address this important potential limitation, we conducted an additional sensitivity analysis that utilized a set of questions from the NESARC Wave 2 assessment that were not utilized in the development of the DRS. Specifically, during a different portion of the NESARC Wave 2 interview, participants were also asked, “In your entire life, did you ever attempt suicide?” If participants answered affirmatively, they were then also asked “How old were you the first time?” and “How old were you the most recent time?” [22]. While some researchers interested in developing predictive models of suicide attempts within NESARC have defined their primary outcome as the most recent attempt having occurred within 3 years of participants’ age at Wave 2 [22], we instead elected to utilize the question from the Low Mood portion of the Wave 2 interview to develop the DRS because (1) it specifically inquired about new instances of suicide attempts occurring since the Wave 1 interview; (2) the number of days between Wave 1 and Wave 2 was variable, making it impossible to definitively determine if a suicide attempt occurring within 3 years of participants’ age at the time of the Wave 2 interview actually occurred after the Wave 1 assessment; and (3) the suicide attempt question from the Low Mood portion of the interview actually identified a greater number of prospective suicide attempts (241 versus 222).

To assess the potential impact of our choice to use the suicide attempt question from the Low Mood portion of the NESARC as our primary outcome variable, we calculated ROC curves for the DRS in the combined NESARC validation cohort (i.e., NESARC 3 and NESARC 4, combined N = 17,244) utilizing 3 different outcomes: (a) suicide attempts occurring since the last interview based on the Low Mood portion of the Wave 2 interview (i.e., our original operational definition which we used to develop the DRS), which resulted in 122 suicide attempt cases and 17,122 controls in the combined NESARC validation cohort; AUC = 0.92, 0.89 to 0.94; (b) most recent suicide attempt having occurred within 3 years of participants’ age at Wave 2 [22], which resulted in 121 suicide attempt cases and 17,123 controls, AUC = 0.91, 0.88 to 0.93; and (c) suicide attempts identified by either method, which resulted in 175 suicide attempt cases and 17,069 controls, AUC = 0.91, 0.88 to 0.93. Thus, the AUC values for the DRS across these 3 different suicide attempt definitions were remarkably similar, ranging from 0.91 to 0.92, and had highly overlapping 95% CIs. It should also be noted that suicide attempts were not assessed within the context of mood disorders in either REHAB or VALOR. Thus, our initial findings suggest that the DRS is similarly effective at identifying risk for future suicide attempts that occur outside of the context of mood disorders.

Comparison of the DRS with a logistic regression–derived risk score

During the peer review process, while noting the attractiveness of the simplistic scoring approach we utilized because of the ease with which it could be manually calculated by clinicians, reviewers requested that we also explore whether more specific weights derived directly from a logistic regression model might further improve the predictive performance of the DRS within the NESARC sample. Accordingly, we conducted an additional logistic regression on the 23 variables used to calculate the DRS in the combined NESARC development cohort (i.e., NESARC 1 and NESARC 2, combined N = 17,397). As can be seen in Table E in S1 File, the variables most strongly associated with prospective suicide attempts in the logistic regression model included BPD (AOR = 5.87, 3.65 to 9.43, p < 0.001), lifetime NSSI (AOR = 3.52, 2.12 to 5.85, p < 0.001), LGBTQ status (AOR = 3.18, 1.66 to 6.11, p = 0.001), lifetime psychiatric hospitalization (AOR = 2.58, 1.47 to 4.50, p = 0.001), and poor perceived health (AOR = 2.07, 1.32 to 3.25, p = 0.001). In contrast, the variables with the weakest association with prospective suicide attempts in the logistic regression model included psychiatric hospitalization in the past year, lifetime mood disorder, weekly binge drinking, lower income, and having less than a high school education (all p’s > 0.45). Next, we used the regression coefficients from the logistic regression model conducted in the NESARC development cohort as risk score weights for the 23 variables. We then calculated ROC curves to compare the predictive validity of the DRS with the logistic regression–derived risk score in the NESARC validation cohort (i.e., NESARC 3 and NESARC 4, combined N = 17,244), which included 122 suicide attempt cases and 17,122 controls. This analysis revealed that the DRS (AUC = 0.92, 0.89 to 0.94) performed quite similarly to the logistic regression–derived score (AUC = 0.91, 0.88 to 0.94), despite using a much simpler scoring approach. In addition, DeLong test confirmed that the AUCs for the 2 models were not significantly different (z = 0.66, p = 0.51), providing additional support for our overall approach to measurement development, which maximized predictive utility while still providing clinicians with a simple scoring approach that can be calculated by hand.

Comparison with the SAD PERSONS scale

During the peer review process, reviewers also requested that we directly compare the DRS with the SPS [12] within the same dataset. The SPS is an acronym and mnemonic device developed by Patterson and colleagues in 1983 [12] to guide assessment of suicide risk. The scale is widely utilized [11] and was specifically developed to teach medical students how to assess suicide risk [12]. Patients are assigned 1 point for each of the 10 risk factors that are deemed to be present by the clinician at the time of the assessment. The specific risk factors to be assessed include: Sex, Age, Depression, Previous attempt, Ethanol abuse, Rational thinking loss, Social supports lacking, Organized plan, No spouse, and Sickness [12]. We developed scoring for the SPS in the NESARC sample since this study included reasonable assessments for 9 of the 10 SPS items (see Table F in S1 File for details on scoring procedures for the SPS in NESARC). The only item that was not directly assessed in NESARC was “Organized plan,” for which we substituted lifetime suicidal ideation, which, notably, had the second highest overall bivariate AUC across the development samples (average AUC = 0.72; see Table A in S1 File). The SPS (M = 2.8; SD = 1.6; range: 0 to 10) exhibited an AUC of 0.74 (0.69 to 0.79) in the combined NESARC validation cohort (combined N = 17,244), which was a better performance than it has shown in some prior studies [13]; however, this value was only slightly better than the AUC for lifetime suicidal ideation by itself in the NESARC validation cohort (AUC = 0.72, 0.66 to 0.77) and was significantly worse than the AUC for the DRS (AUC = 0.92, z = 8.2, p < 0.0001), indicating that the DRS was significantly better than the SPS at predicting future suicide attempts in the NESARC validation cohort.

Study limitations and future directions

It should, however, also be emphasized that the present findings are based entirely on secondary analyses of rigorously collected, prospective research data. As a result, the degree to which the current findings will generalize to other settings, including clinical settings, is unknown at the present time. Thus, additional prospective research is still needed to validate the DRS in independent samples and to determine how best to assess each of the constructs used to calculate it. Relatedly, it is also important to note that there is presently no empirical support for the DRS in situations in which clinicians rely exclusively on their clinical impressions of patients to calculate the DRS score (as opposed to using standardized instruments to assess each of the 23 DRS constructs). Accordingly, we strongly recommend that clinicians who wish to utilize the DRS in their practice adhere to the guidelines provided in the S1 Durham Risk Score Guide.

Second, as detailed above, several DRS variables (e.g., abuse history) were not assessed during the NESARC Wave 1 assessment. We ultimately felt that the inclusion of these variables in the DRS was warranted, given that our explicit goal was to make the best possible clinical tool. Additionally, a sensitivity analysis revealed that the DRS continued to perform quite well when calculated exclusively from Wave 1 items; however, we fully recognize that this remains a limitation of the present work and that more work is still needed to verify the utility of these variables in studies where they are assessed prospectively.

Third, it is unclear how well the findings from the present research, which are derived entirely from participants who consented to participate in longitudinal research studies, might generalize to individuals undergoing clinical assessments or seeking treatment for mental health issues. One might expect that some individuals undergoing suicide risk evaluations would be less willing to disclose potential risk factors, including current and prior history of suicidal thoughts and behaviors. Of course, the latter concern also applies to virtually all clinical suicide risk assessments currently available, as nearly all such assessments rely on participants’ willingness to disclose suicidal thoughts and plans. Additionally, a strength of the DRS is that much of the score is derived from demographic and psychopathology-based risk factors, which should, theoretically, make it more robust to underreporting than traditional risk assessments that often rely exclusively on participants’ reports of suicidal thoughts and behaviors.

Fourth, we recognize that the DRS contains many items and may not be practical to adminster in some settings. For this reason, we are actively working to develop an abbreviated version of the measure. Fifth, although we are strong proponents of multiple imputation and maximum likelihood estimation methods to handle missingness in most situations, we elected to treat missing data as absent (i.e., “0”) in the calculation of DRS scores in this study because this approach best reflects real-world clinical practice. Further, we strongly believe that the benefits of a raw suicide attempt risk score that can be calculated and interpreted in virtually any clinical situation far outweigh the benefits of using more state-of-the-art approaches to handling missingness.

Finally, far more work is needed to develop more accurate short-term risk models (e.g., 1-week or 1-month models). Although the DRS is unable to accomplish this goal, we believe that identification of long-term suicide attempt risk is a key first step toward developing more accurate acute risk models. Specifically, consistent with fluid vulnerability theory [30] and diathesis–stress models more generally, we hypothesize that individuals who score highly on the DRS will have higher “set points” and will be more likely to attempt suicide when faced with highly stressful situations than individuals with lower scores. While such work was beyond the scope of the present study, we are hopeful that future prospective studies in this important area of inquiry will lead to improved prediction of acute risk, which, unfortunately, remains woefully inadequate at the present time.

Clinical implications

Although clinicians must ultimately determine which treatments are most appropriate for their patients, consideration should be given to the idea of developing a safety plan [69] with any patient who scores ≥6 on the DRS (i.e., moderate-risk group status or higher), as this brief intervention has been shown to significantly reduce the occurrence of future suicidal behavior [70]. Moreover, as noted above, recent research by Ross and colleagues [66] indicates that the specificity (86%), sensitivity (82%) and PPV values (4%) corresponding to a cut score of 6 or higher on the DRS exceed the threshold accuracy values necessary for cost-effective implementation of suicide risk prediction with safety planning and follow-up. Consideration should also be given to ensuring that patients in the highest-risk groups have access to more intensive and long-term cognitive behavioral treatment approaches that have also been shown to reduce the occurrence of suicidal behavior [71–72]. Importantly, a cut score of 9 or greater on the DRS (corresponding to high-risk group status or approximately top 5% of scores) produces specificity (96%), sensitivity (58%), and PPV (10%) values that exceed the threshold accuracy values identified by Ross and colleagues [66] as necessary for suicide risk prediction to be combined with cognitive behavioral therapy for suicide prevention to become cost-effective from a healthcare sector perspective.

On the other hand, clinicians must also recognize that the DRS does not assess acute risk, as the latter determination requires in-depth review of current suicidal ideation, intent, plans, feasibility, access to means, and current stressors (among others). As such, the DRS should never be used as the sole basis to determine imminent suicide risk or need for civil commitment. Instead, individuals endorsing recent suicidal ideation or behaviors should always be assessed for intent and other indicators of acute risk not included in the DRS. Where possible, we recommend that the DRS be integrated with existing assessment and intake practices, as most DRS items are routinely assessed by mental health clinicians. Constructs not already assessed could likely be added with relatively little additional burden to mental health clinicians and clients. An additional advantage of making the DRS part of routine practice is that mandatory checklists have been shown to increase the occurrence of risk-appropriate treatment while simultaneously decreasing healthcare disparities [73]. Notably, there are documented disparities in the application of mental health and suicide risk assessments [74] that could potentially be eliminated with a DRS-based clinical decision support model.

A final clinical consideration concerns our finding that 42% of all prospective suicide attempts during the follow-up period occurred among individuals who reported no lifetime history of suicidal thoughts or behavior at baseline. This important finding speaks to the tremendous challenges that clinicians routinely face when attempting to stratify patients’ risk for future suicidal behavior, as the vast majority of current suicide risk screens primarily rely on patients’ current endorsement of suicidal thoughts and behaviors. In contrast, the approach taken in the development of the DRS has been to focus on a diverse array of longitudinal risk factors. Importantly, both the DRS total score and risk group status performed well among individuals with and without a lifetime history of suicidal thoughts or behaviors, which we believe provides strong support for our general approach to instrument development.

Disclaimers

The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the US Department of Veterans Affairs, the US Department of Defense, the US government, or any of the other affiliated institutions.