Introduction

Suicide is a major public health concern globally. Predicting suicidal behavior is challenging both at the population level and among high-risk groups. The accuracy of predicting suicidal behavior based on clinical judgment varies considerably across clinicians. Risk factors known to be strongly associated with suicidal behaviors are weak predictors on their own [1]. One meta-analysis assessing the sensitivity and specificity of 15 different instruments for suicide and suicide attempt concluded that none of these instruments provided sufficient diagnostic accuracy defined by the authors (i.e., 80% for sensitivity and 50% for specificity) [2]. However, using a lower threshold for discrimination measures, it is possible for a prediction model to achieve the specified diagnostic accuracy, though most likely at the cost of a reduced positive predictive value (PPV). Another meta-analysis assessing the performance of previously reported psychological scales, biological tests, and third-generation scales derived from statistical modeling (mostly using conventional multivariable regression) for prediction of suicidal behavior [3]. The authors, who did not synthesize performance metrics other than PPV, reported a pooled PPV of 39% for the third-generation scales for predicting suicide attempts/deaths. One potential explanation for the modest predictive performance is that the data used for previous model development did not contain enough information for making accurate prediction. It is also possible that prediction of suicidal behavior is too complex to be based on a few simplified theoretical hypotheses [4].

Although currently not feasible to implement, difficult to understand by clinicians, and lacking transparency [5], machine learning algorithms have been applied to large-scale data such as electronic medical records for predicting suicidal behavior. In machine learning analytics, selecting candidate predictors may benefit from established theories and clinical expertise. When given access to large amounts of new data, machine learning may serve as an efficient and flexible approach to exploring the predictive potential of the new data. Meanwhile, machine learning algorithms usually identify far more complex data patterns than conventional methods, though at the cost of decreased interpretability [4]. Belsher and colleagues systematically reviewed 64 machine learning–based prediction models for suicide and suicide attempts in 17 studies and found that, despite good overall discrimination accuracy, the PPVs remained low, with inadequate information on negative predictive value (NPV) and calibration [6]. In their subsequent simulation analyses, they demonstrated that the achievable PPV was limited by the rarity of suicide even when sensitivity and specificity are hypothetically set to be nearly perfect. They thus recommended future research focus on predicting more common outcomes such as suicide attempts [6].

Most prior studies on predicting suicide or suicide attempts have been limited by small sample sizes. Very few of them provided a comprehensive report of model discrimination, including sensitivity, specificity, PPV, and NPV, as well as calibration. Only a single type of model (e.g., random forest model or regression model) was selected in most studies. Data from multiple Swedish national registers have been used to develop a multivariable regression model for predicting suicide among patients with schizophrenia spectrum disorders and bipolar disorder [7]. To date, the data have never been combined with machine learning to predict suicidal behavior in the setting of psychiatric specialty care.

In this study, we aimed to examine the achievable performance of models trained by several machine learning algorithms using the Swedish registry data. We developed prognostic prediction models for suicide attempt/death within 90 and 30 days following an in-/outpatient visit to psychiatric specialty care, using predictors generated via linkage between multiple Swedish national registers.

Methods

Results

Among 541,300 eligible visits to psychiatric specialty care, 18,682 (3.45%) were followed by suicidal outcomes within 90 days and 9,099 (1.68%) within 30 days. Descriptive characteristics of the entire study sample are shown in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Baseline characteristics of 541,300 eligible visits by 126,205 patients to psychiatric specialty care during 2011 and 2012.

https://doi.org/10.1371/journal.pmed.1003416.t001

Model discrimination

The models for predicting suicide attempt/death within 90 and 30 days following a visit to psychiatric specialty care demonstrated good discrimination accuracy. The test AUCs were 0.88 (95% CI: 0.87 to 0.89) and 0.89 (95% CI = 0.88 to 0.90), respectively (Fig 1). At the 95th percentile risk threshold, the sensitivities were 47.2% and 52.9%, the specificities were 96.6% and 95.9%, the PPVs were 34.9% and 18.7%, and the NPVs were 97.9% and 99.1% (Table 2 and S7 Table). Table 2 also shows sensitivity, specificity, and predictive values over a series of risk thresholds.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1.

ROC curves illustrating model discrimination accuracy in the test set for predicting suicide attempt/death within 90 (A) and 30 days (B) following a visit to psychiatric specialty care during 2011 and 2012. The figure was based on the discrimination accuracy of the ensemble models. The solid line in brown represents the ROC curves achieved by the models. The dotted line in black represents the ROC curves when AUC equals 50%. AUC, area under the receiver operating characteristic curves; ROC, receiver operating characteristic.

https://doi.org/10.1371/journal.pmed.1003416.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Model performance metrics at various risk thresholds for predicting suicide attempt/death within 90 and 30 days following a visit to psychiatric specialty care during 2011 and 2012.

https://doi.org/10.1371/journal.pmed.1003416.t002

Model calibration

Brier scores were estimated to be 0.028 and 0.015 (both close to 0) for the models predicting outcome events within 90 days and 30 days, respectively, indicating good model calibration. The calibration plots (Fig 2) further illustrate high agreement between the observed proportion of positives and mean predicted risk of the outcomes. More details can be found in S9 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2.

Calibration plots comparing observed proportion of positives and mean predicted risk suicide attempt/death within 90 (A) and 30 days (B) following a visit to psychiatric specialty care during 2011 and 2012. The figure was based on the calibration of the ensemble models. Each solid dot in blue represents the observed proportion of index visits followed by a suicidal event in a bin of sample (observed proportion of positives [suicidal events]) against the mean predicted risk in the same bin. More details can be found in S9 Table. The rug plot in pink represents the distribution of the study sample across different predicted risk levels.

https://doi.org/10.1371/journal.pmed.1003416.g002

Learning curve analyses: bias versus variance

Fig 3 illustrates the learning curves for assessing the bias and variance trade-off. As the training sample size gradually increases, the 2 curves representing training and validation AUCs end up very close to each other and converge at AUCs of 0.88 and 0.89 for the models separately predicting outcome events within 90 and 30 days, suggesting relatively low bias (relatively high AUCs) and low variance (eventual convergence). Because the validation curve is no longer increasing with increased training sample size, future improvements to the model may require more informative predictors and higher model capacity rather than a larger sample size. The convergence of the training and validation AUCs indicates no model overfitting.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3.

Learning curves illustrating bias and variance trade-off in the training set for predicting suicide attempt/death within 90 (A) and 30 days (B) following a visit to psychiatric specialty care during 2011 and 2012. The figure was based on the calibration of the ensemble models. AUC, area under the receiver operating characteristic curve.

https://doi.org/10.1371/journal.pmed.1003416.g003

Importance of predictors

Table 3 shows the top 30 predictors with the highest importance based on the best elastic net penalized logistic regression, random forest model, and gradient boosting models, as well as a substantial overlap in top predictors between these models. In general, temporally close predictors tended to be ranked higher than temporally remote predictors. Intentional self-harm during the past 1 year (i.e., within 1 month, 1 to 3 months, 3 to 6 months, or 6 to 12 months), unplanned visit to psychiatric specialty care service during the past 1 to 3 months, diagnosis of borderline personality disorder during the past 3 months (i.e., within 1 month or 1 to 3 months), and diagnosis of depressive disorder during the past month, recent dispensation of antidepressants (i.e., 3 to 6 months), anxiolytics (i.e., 6 to 12 months), benzodiazepines (i.e., within 1 month or 6 to 12 months), and antipsychotics (i.e., 1 to 3 years) were ranked as top predictors. In addition, prior intentional self-harm by poisoning or sharp object, family history of suicide attempt, family history of substance use disorder, and family history borderline personality disorder were also ranked as top predictors by more than 1 model. The intercepts and coefficients of the elastic net penalized logistic regression models can be found in S10 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Predictors ranked top 30 by the best models of elastic net penalized logistic regression, random forest, and gradient boosting.

https://doi.org/10.1371/journal.pmed.1003416.t003

Discussion

To the best of our knowledge, this is the first study using machine learning to determine the potential of the Swedish national registry data for relatively short-term prediction of suicidal behavior in the general psychiatric specialty care. Based on ensemble learning of elastic net penalized logistic regression, random forest, gradient boosting, and a neural network, the final models achieved both good discrimination (AUC was 0.88 [0.87 to 0.89] for the 90-day outcome and 0.89 [0.88–0.90] for the 30-day outcome) and calibration (Brier score was 0.028 for the 90-day outcome and 0.015 for the 30-day outcome).

The AUCs achieved by our models were higher than those of prior studies that predicted suicidal behavior within the 90-/30-day windows in the review of Belsher and colleagues [6]. One model in a prior study among the United States army soldiers with suicidal ideations demonstrated higher AUC (0.93) [32]. The authors of the study, however, did not specify the time window of the predicted outcome. They used cross-validation rather than a separate dataset for the final model evaluation, which tended to overestimate the AUC as indicated by the previous review and simulations [33,34].

To date, there is a lack of agreement on what risk threshold would signal sufficient clinical utility of a prediction model for deployment in clinics. We therefore reported 4 additional model metrics at varying risk thresholds instead of focusing on 1 single preselected threshold. In the current study, at the 95th percentile risk threshold, the model would correctly identify approximately half of all suicide attempts/deaths within 90 days (sensitivity 47.2%); among those psychiatric visits that were predicted to be at high risk, around one-third were actually followed by suicidal events within 90 days (i.e., PPV 34.7%). Although a higher sensitivity could be achieved at a lower threshold, this would be at the cost of a reduced PPV. At the 80th percentile risk threshold, nearly 80% of all suicide attempts/deaths within 90 days would be correctly identified (sensitivity 77.4%); among those visits that were predicted to be at high risk, 1 in 7 was followed by suicidal events within 90 days (PPV 14.3%). Meanwhile, most visits without suicidal events within 90 days would be correctly identified given the high specificity estimates (96.6% and 82.2% at the 95th and 80th percentile risk thresholds, respectively), while most visits predicted to be low risk were not followed by suicidal events given the high NPV (97.9% and 99.0% at the 95th and 80th percentile risk thresholds, respectively). Selecting a reasonable risk threshold for predictive values depends on the resources available for subsequent intervention strategies and their implications for individuals and services. When effective intervention strategies are taken into consideration, model performance metrics at other risk thresholds may also be informative. Future research on cost-effective interventions with negligible or no adverse effects are warranted (e.g., increased frequency of follow-up), as such interventions could be allocated to false-positive patients using models with low PPVs. In addition, prior studies tended to put more emphasis on the role of PPV in assisting clinical decision-making, although NPV might be useful in confirming clinical assessment of low-risk group and helpful in optimizing the allocation of healthcare resource. One prior study, using the Swedish registry data collected during 2001 to 2008 and multivariable regression, developed and validated a prediction model for suicide within 1 year after discharge among in- and outpatients diagnosed with schizophrenia spectrum or bipolar disorder [7]. The final model achieved an AUC of 0.71 on the final validation set. The study prespecified a risk threshold of 1% (i.e., 99th percentile risk threshold), which was close to the prevalence of the predicted outcome (approximately 0.6%), for evaluating the model metrics, although concluded that this risk threshold should not be used clinically, and probability scores were more informative. At the 1% threshold, the sensitivity, specificity, PPV, and NPV were estimated to be 55%, 75%, 2%, and 99%, respectively. Despite a very low PPV, the model achieved a high NPV. Similarly, in the current study, at the risk thresholds close to the prevalence estimates of the predicted outcomes, the NPVs were also high for suicide attempt/death with 90 days (97.8%) and 30 days (98.8%). However, it is difficult to directly compare the models from the 2 studies, given the differences in definition of the predicted outcome (suicide death versus suicide attempt or death) and time window of interest between the studies. Currently, it is unclear under which circumstances a seemingly very high NPV may have clinical utility for an outcome with a low prevalence. Simulation studies are therefore required to determine how NPV varies with the values of PPV, outcome prevalence, and potential cost of intervention. Moreover, screening out on the basis of high NPVs will be limited to the outcome of interest and will not be appropriate if those individuals are at increased risk of other adverse outcomes (e.g., accidents). This suggests that if tools are used in this way, clinicians need to consider risks for other outcomes before these individuals are not further assessed or considered for treatment.

In the current study, the difference in predictive performance between the initial 4 types of models, namely elastic net penalized logistic regression, random forest, gradient boosting, and a neural network, was very small. This is consistent with the findings in the review by Belsher and colleagues suggesting that no 1 model seems clearly better than another [6]. Future research using the same predictors as in our study may employ elastic net penalized logistic regression only to obtain a relatively better model interpretability at a limited loss of predictive performance. The learning curves suggest that more informative predictors and higher model capacity are likely required to further improve the predictive power. Informative predictors could come from creating transformations of existing predictors (e.g., the frequency of self-harm over a certain time period) or incorporating completely novel predictors such as data from primary medical care services, clinical documentation text, audio and video data on clinical interview, vital physiological parameters continuously monitored via wearable devices. Novel and previously untested data may open opportunities for deep learning analytics to improve prediction of suicidal behavior through identifying highly complex data patterns [35,36]. Deep learning analytics may also improve model capacity by creating a better representation of the predictors used in the current study [37].

There was a substantial overlap in top-ranked predictors between different models. While borderline personality disorder, substance use disorder, depression, dispensed benzodiazepines and/or antidepressants have already been identified as risk factors for suicide [38,39], the timing of these factors may play a vital role in predicting subsequent suicidal behavior [40]. Our study showed that temporally close predictors tended to be ranked higher than distal ones. These results call for more research in the timing of diagnoses of psychiatric disorders and the use of psychotropic medications in relation to suicidal behavior. Using the Danish registry data, a recent study predicting sex-specific suicide risk reported that diagnoses occurring long (e.g., 48 months) before suicide were more important for suicide prediction than diagnoses occurring shortly (e.g., 6 months) before suicide [41]. Since the authors defined the predictors using time of suicide, which could not be known beforehand, the models would not be implementable in clinical systems. One Swedish registry-based cohort study of 48,649 individuals admitted to hospital after suicide attempt found that prior hanging, drowning, firearms or explosives, jumping from a height, or gassing at suicide attempt better predicted subsequent suicide than poisoning and cutting [42]. Our study, however, showed that poisoning and cutting were ranked higher than other methods when predicting a broadly defined behavior including both suicide attempt and suicide. As expected [43], family history of several psychiatric disorders (i.e., anxiety disorders, borderline personality disorder, major depressive disorder, substance use disorder) were also among the top predictors, whereas family history of somatic disorders were not. It should be noted, however, that in predictive modeling, the importance of a predictor reflects the extent to which permuting the value of the predictor will increase prediction error [44]. Unlike causal risk factors, predictors with higher importance may or may not have a direct impact on the outcome. Highly correlated predictors, differing in model-based importance scores, may have very similar univariate predictive ability but the one that is less predictive would be given low importance since its importance value is conditional on the highly correlated predictor being in the model. The coefficients in the elastic net penalized logistic regression, unlike in ordinary logistic regression, were shrunk in magnitude. Although the size of coefficients could be used for ranking predictor importance, it does not represent the effect size of a specific predictor on suicidal behavior. Moreover, the sign (positive/negative) of coefficients should not be interpreted as an increase or decrease in risk of suicidal behavior. This is because, in predictive modeling, the predictors are not necessarily causal risk factors of the predicted outcome and the direction of effect of a specific predictor is influenced by other model selected predictors that are not necessarily confounders. Of more relevance in predictive modeling is the overall predictive performance rather than individual model coefficients. Accurate estimation of the magnitude and direction of risk factors requires different research designs, such as cohort or cross-control investigations. When candidate predictors were restricted to those related to patient’s history of diseases and dispensed medications only, the AUCs were only slightly worse than those achieved by the models using all predictors obtained via complex linkages among registers, suggesting limited incremental value of the predictors from other registers than the National Patient and the Prescribed Drug Registers in predicting suicidal behavior. The results have pragmatic implications, because it is more feasible to integrate data from within the electronic medical records system than it is to create complex linkages among registries.

Our study is subject to several limitations. To ensure reasonable quality of data on both individual health and social economic information, as well as family history of disease and crime, the sample was limited to patients who were adults at the visit and covered by the Swedish Medical Birth Register. By the end of year 2012, the oldest in the study population were 39 years old. The derived models therefore are less likely to generalize well to adult patients older than 40 years or to children and adolescents. The candidate predictors were limited by the data sources. The derived models are not clinically based as hospital services are not in a position to link disparate registers. Our outcomes have limited validation. Nevertheless, according to an external review on the validity of diagnoses in the Swedish national inpatient register [13], the PPVs of the register-based diagnostic codes varied between 85% and 95%. Notably, the PPV of injury, including both accidental injury and self-injury, was 95%. On the basis of this, we think that the PPV of our outcome is likely to be high, despite a lack of specific validation study. Our definition of suicidal behavior was based on the ICD-10 criteria, which does not make distinction between suicidal and nonsuicidal self-harm. Although this definition has been widely used in prior research, whether nonsuicidal self-harm represents actual suicide risk remains controversial. A certain amount of nonsuicidal self-harm events were labeled as positive outcome events, which may have led to an overestimation of the PPVs. On the other hand, underestimation of the PPVs is also possible as planned visits with a recorded self-harm were not labeled as positive outcome events. When defining the outcome, we assumed that unplanned inpatient or outpatient visits after the index hospitalization with a recorded self-harm were incident suicidal events. This was based on the selection of the study sample, which was restricted to patients who, at the index visit, received a primary diagnosis with an ICD-10 code ranging from F00 to F99. This means that the index visits were not primarily due to suicide attempt/self-harm (which has a different ICD code, i.e., ICD-10: X60–X84 and Y10-Y34), and thus, the identified suicidal events after the index visit were very unlikely to be the same visit as the index visit. Future validation studies could examine how accurately identified events in the Patient Register represent true incident suicidal events. The preliminary selection of 425 candidate predictors and their time windows was somewhat arbitrary. In theory, many more candidate predictors can be generated from the registry data or via transformation of existing ones. As a result, it is unclear whether the achieved model performance in the current study is the best achievable performance. Our models have not yet been externally validated, and thus, the generalizability of the models to other populations remains unknown. In future work, guidelines will be critical in improving transparency and reproducibility of research in predictive modeling and should be followed once they are finalized.

In our study, the models were derived to predict suicidal behaviors rather than the rare outcome of suicide deaths. In relation to deaths, the model performance would be quite different with higher specificities and lower sensitivities, and the calibration would be poor. As some metrics of discrimination will be low, 1 promising approach would be to solely use probability scores rather than risk thresholds to investigate suicide deaths [7]. Future research could compare ordinary logistic regression models with machine learning models, and in particular the former can focus on the top-ranked predictors identified in this study.

Conclusions

By combining the ensemble method of multiple machine learning algorithms and high-quality data solely from the Swedish registers, we developed prognostic models to predict short-term suicide attempt/death with good discrimination and calibration. Whether novel predictors can improve predictive performance requires further investigation.

Supporting information