Analysis of Hospitals Switching From a “Danger to Self” Question to Universal Columbia‐Suicide Severity Rating Scale Screening: Impact on Screenings, Identification of Suicide Risk, and Documented Psychiatric Care

Suicide is a pervasive and growing public health issue in the United States (1, 2), yet screening for suicide risk remains limited. Suicide risk screening is more common among individuals with mental health disorders who are known to have increased suicide risk (3, 4), with depressive and anxiety disorders being the most common among individuals who die by suicide (4). Major Depressive Disorder (MDD) has a high lifetime prevalence of 20.6% in the general population (5) and is associated with suicidality (6, 7). Coordinated efforts to improve earlier identification of people with suicide risk are urgently needed in order to connect them with appropriate care and prevent suicide deaths (8). Research indicates that 83% of individuals who died by suicide received healthcare in the year prior to their death (9, 10), yet many of these people are not being identified as at‐risk. These statistics represent a missed opportunity for intervention. Insufficient detection, monitoring, and follow‐up interventions in healthcare settings contribute to suicide deaths (11, 12, 13). Effective interventions such as safety planning and psychotherapy exist (14, 15), but are not consistently available.

Many people with suicide risk present for care in acute care settings, including emergency departments (EDs) (16, 17). The ED‐SAFE study demonstrated the effectiveness of universal suicide screening in EDs and brief interventions to reduce suicide attempts (18, 19). In 2018, The Joint Commission revised its National Patient Safety Goal requirements by standardizing suicide screening in behavioral health acute care settings and recommending use of validated suicide screening tools such as the C‐SSRS (20, 21). The C‐SSRS screen version (triage version) is a 6‐item questionnaire which detects suicide risk and severity and immediacy of suicide risk (22, 23), and its validity and feasibility has been established in emergency, psychiatric department, and general inpatient departments (24, 25, 26). While a number of health systems have adopted C‐SSRS screening (27), there is little mention of prior screening methods except recognition of the need for a standardized approach (22, 23). It is unknown whether screening with C‐SSRS compared to pre‐existing un‐validated screening questions has led to measurable changes in important population health outcomes.

Whereas the C‐SSRS is often administered based on evidence of suicide risk or existing mental health needs (24, 25, 28), this retrospective study examined the implementation of the C‐SSRS within the general population. Specifically, this study evaluated the impact of switching from a single “Danger to Self” (DTS) screening question to the C‐SSRS questionnaire for all patients seen in its 23 acute care hospitals with respect to three outcomes: rates of (1) screening, (2) positive screenings for suicide risk, and (3) documentation of follow‐up psychiatric care within the electronic health record (EHR).

Setting and Screening Methods

This research took place at Sutter Health, a large integrated healthcare delivery system in northern California which cares for approximately 3.5 million people each year in 100+ ambulatory clinics, 23 acute‐care hospitals, four acute care behavioral health centers, and 6 ambulatory behavioral health clinics. Prior to the C‐SSRS implementation, clinicians answered one question in the EHR about whether “DTS,” such as suicidal ideation or behavior or other self‐harm indicators were observed or expressed from a patient. DTS was assessed by RNs/clinicians and was required as part of the standard admission process across Sutter Health hospitals. DTS was based on clinical judgment and the wording of the questions asked and exactly when it was assessed varied across the hospitals.

Implementation of the C‐SSRS in these acute care hospitals and the population identified with suicide risk by C‐SSRS has been fully described elsewhere (29). Sutter Health first piloted use of the screen version (triage version) of the C‐SSRS in one general ED and two acute care behavioral health departments, then launched system‐wide standardized use of the C‐SSRS on July 1, 2019, in all acute care facilities, replacing the DTS question, and integrating C‐SSRS screening questions into the EHR. A workgroup of system stakeholders coordinated the implementation and met regularly. This group also organized standardized 20‐min C‐SSRS trainings conducted online or in person for approximately 9000 acute care registered nurses, including training on administration and entry in the EHR. The C‐SSRS is administered verbally, primarily by nurses, in EDs and inpatient acute care settings to all patients 10 years or older. With guidance from the developers of the C‐SSRS, individuals' responses were assigned to either low, moderate, or high risk categories (Online Supplement Table S1), along with accompanying practice recommendations. Low risk recommendations included considering mental health referrals, and for moderate or high risk included further assessment, immediate provider notification and mental health consultations, and additional safety precautions for high risk individuals.

METHODS

This EHR‐based observational cohort study analyzed changes in rates of screenings, positive screenings, and documented psychiatric care associated with the implementation of the C‐SSRS. The study period included a 24‐month pre‐launch “pre‐period” (July 1, 2017 to June 30, 2019) and 18‐month “post‐period” (July 1, 2019 to December 31, 2020). The cohort of patients seen included unique adults (age ≥18) in each time period with an index encounter at any of the 23 hospitals. For patients with multiple hospital encounters within the time period, their first encounter with a completed screening served as the index encounter, or if they were never screened, their first hospital encounter served as the index encounter. This study was reviewed and approved by Sutter Health's institutional review board.

Primary Outcomes

In the pre‐period, screening was measured by presence of response to the DTS question, and it was not possible to measure level of suicide risk. In the post‐period, screening was measured by presence of a complete C‐SSRS questionnaire. If patients had multiple screenings during a hospitalization, the first complete screening was used. Positive screening was defined in the pre‐period as having DTS identified and in the post‐period as being identified with low, moderate, or high risk by C‐SSRS.

A composite variable captured EHR documentation of any psychiatric care within 90 days of the index encounter, including transfer to psychiatric unit, discharge to psychiatric hospital, or behavioral health consultation/referral. Transfers and discharges to both Sutter and non‐Sutter hospitals were retrieved from hospital discharge data. Psychiatric consultations were limited to care within Sutter's acute and ambulatory care system. Referrals to psychiatric care were predominately within Sutter's system, with the exception of referrals to a vendor responsible for coordinating behavioral health for ambulatory care patients in specific geographic areas. Other subsequent care measured included patient hospitalization and length of stay at the time of the index encounter, and additional hospitalizations with mental health diagnoses recorded, behavioral health acute care hospitalizations, ED visits, and C‐SSRS or DTS screenings recorded.

Covariates

Other measures included patient information retrieved from the EHR in the 12 months prior to and including the index date. Sociodemographic characteristics included age, sex, race, ethnicity, language spoken, marital status, median household income for patient's postal code, and insurance type. Healthcare utilization included type of encounter (ED, inpatient, or observation), department of index encounter, and number of prior primary care and ED encounters. Clinical characteristics included diagnosis of type 2 diabetes, hypertension, cancer, congestive heart failure, chronic pulmonary disease, and Charlson Comorbidity Index of 1 or more (30). Mental health diagnoses included MDD, anxiety disorder, depressive disorders, substance abuse disorder, bipolar spectrum disorder, schizophrenia spectrum disorder, Attention‐Deficit/Hyperactivity Disorder, autism, gender dysphoria, dementia, eating disorder, conduct/disruptive disorder, personality disorder, and were defined with International Classification of Diseases, Tenth Revision (ICD‐10) codes used in the Mental Health Research Network (31, 32). Prior suicide ideation was measured by the presence of ICD‐10 codes (Online Supplement Table S2).

Statistical Analysis

Descriptive statistics were used to describe population characteristics and primary outcomes in the pre‐period and post‐period. Subgroup analyses were conducted for patients with: (1) evidence of MDD, and (2) moderate or high risk identified by C‐SSRS. Patients with C‐SSRS moderate or high risk were grouped for analysis based on the logic that they most urgently need psychiatric intervention, and individuals identified with low suicide risk via C‐SSRS would be unlikely to screen positive using the DTS question which was intended to capture immediate danger. Data were analyzed using standard tests (Wilcoxon rank‐sum tests for continuous variables and χ² tests for categorical variables) with an alpha of 0.05 (two‐sided) as the level of significance. Analyses were performed using SAS version 9.4.

As this study analyzed simultaneous implementation of the C‐SSRS at all hospitals without randomization, interrupted time series (ITS) analyses were conducted on monthly outcomes from July 2017 to December 2020 to assess the longitudinal effects of the C‐SSRS implementation on the primary outcomes. ITS considered an expected trend during the pre‐period and the trend observed in the post‐period and identified changes in the trend between time periods to evaluate the effectiveness of C‐SSRS implementation (33, 34). Scatter plots of the time series were also created to visualize trends and seasonal patterns.

Multiple ITS analyses were conducted for each outcome and for each group. For the outcome Screening Rate, two models were run separately for the overall patients seen and for the subgroup with MDD diagnosis. For the outcome Positive Screening Rate, one model was created for comparing the rate of positive screenings among patients screened, another for the MDD subgroup who were screened, and one model for positive DTS screening pre‐period and C‐SSRS moderate or high risk post period among patients screened. For the outcome Documented Psychiatric Care Rate, three separate models were run for all patients identified with positive screenings, those with positive DTS screening pre‐period and C‐SSRS moderate or high risk post‐period, and those with MDD and positive screenings.

RESULTS

798,653 unique individuals were seen in the pre‐period and 595,915 in the post‐period (average per month: 33,277 pre‐ vs. 33,106 post‐period) (Table 1). The population (pre vs. post) had a mean age of 48.8 versus 48.7, was 57.2% versus 56.5% female, and both time periods had similar distributions by race and ethnicity: Non‐Hispanic White 49.8% versus 48.3%, Non‐Hispanic Black 11.3% versus 11.3%, Hispanic 21.9% versus 23.0%, and Asian 9.6% versus 9.4%. Patient characteristics and healthcare utilization were similar in both time periods.

Changes After C‐SSRS Implementation

Screening decreased from 92.8% (740,984 out of 798,653) of patients using the DTS question (pre‐period) to 84.6% (504,015 out of 595,915) using the C‐SSRS (post‐period) (Table 2). The rate of screening positive increased from 1.5% of patients screened in the pre‐period to 2.2% post‐period (p < 0.001), with 1.3% identified as moderate or high risk by C‐SSRS. The overall proportion of patients screening positive out of all patients seen increased from 1.35% (10,791 out of 798,653) in the pre‐period to 1.82% (10,866 out of 595,915) in the post‐period.

During the pre‐period, 64.0% of patients who screened positive had documentation of psychiatric care within 90 days compared to 52.5% (p < 0.001) in the post‐period. Among patients screening positive, rates of transfer or discharge to acute psychiatric care decreased from 40.2% pre‐period to 31.2% post‐period (p < 0.001), rates of referrals to behavioral health providers declined from 30.7% pre‐period to 28% of patients post‐period (p < 0.001), and behavioral health consultations declined from 27.3% to 23.1% (p < 0.001). However, out of all patients seen at the hospital, documentation of psychiatric care within 90 days increased from 0.87% pre‐period (6910 out of 798,653 patients) to 0.96% in the post‐period (5706 out of 595,915 patients).

When comparing patients screening positive by DTS in the pre‐period and patients screening positive with C‐SSRS moderate or high risk in the post‐period, rates of documented psychiatric care increased from 64.0% to 66.4% (p = 0.001), transfers/discharges to acute psychiatric care increased 40.2% to 42.4% (p = 0.005), behavioral health consultations increased from 27.3% to 29.7% (p < 0.001) and referrals to behavioral health providers from 30.7% to 34.4% (p < 0.001) (Additional file 4).

There were 6.7% (53,864 out of 798,653) patients in the pre‐period and 7.0% (41,652 out of 595,915) in the post‐period with a documented MDD diagnosis. Rates of screening, screening positive, and psychiatric care were higher in the MDD subgroup compared to the overall cohort. From pre‐ to post‐period screening of patients with MDD decreased, 95.4% to 89.7% (p < 0.0001), positive screenings increased from 9.2% to 10.8% (p < 0.001), and documented psychiatric care for those screening positive decreased from 70.6% to 67.6% (p = 0.002).

Interrupted Time Series Models

The scatter plot of monthly screening rates indicated a slight monthly increasing trend during the pre‐period but an immediate drop and a slight monthly declining trend in the post‐period following C‐SSRS implementation (Online Supplement Figure S1). The ITS model further supported this evidence showing a decrease of −9.61% (p < 0.01) in the first month after the C‐SSRS was implemented with a monthly trend change of −0.34% (p < 0.01) compared to the pre‐period trend (Table 3). For the subgroup diagnosed with MDD, ITS results showed an immediate decrease of screening rates by −9.4% (p < 0.01) after C‐SSRS implementation, followed by a monthly trend change of −0.19% (p < 0.01) relative to the pre‐period trend.

The scatter plot of monthly rates of positive screenings showed a stable trend during the pre‐period, but an immediate increase after C‐SSRS implementation followed by a subsequent declining trend (Figure 1). The proportion of patients screening positive increased by 1.29% (p < 0.01) in the month following implementation, followed by a slight decrease of −0.02% in the monthly trend compared to the pre‐period trend. For the proportion of patients identified with moderate or high risk in the post‐period, the immediate effect was a slight increase of 0.04%. For patients with diagnosed MDD, the ITS model showed an increase of 6.09% (p < 0.01) in the first month following implementation, relative to the pre‐trend, with a decrease of −0.14% (p < 0.01) in the monthly trend thereafter.

Among patients screening positive, the monthly rate of documented follow‐up psychiatric care appeared stable during pre‐period but varied across different risk levels identified by C‐SSRS during the post‐period (Figure 2). The ITS analyses showed a decrease of −12.86% (p < 0.01) in overall patients with documented psychiatric care in the month following C‐SSRS implementation followed by a similar monthly trend as the pre‐period. However, for patients identified with moderate or high risk in post‐period, the ITS model showed an immediate increase of 4.0% (p < 0.01) followed by a monthly trend decrease of −0.34% (p < 0.01). For the MDD subgroup, there was an immediate decrease of −8.57% in psychiatric care after C‐SSRS implementation with a monthly trend decrease of −0.16%.

DISCUSSION

This study analyzed changes in rates of screening, positive screenings for suicide risk, and documented psychiatric care in 23 hospitals that implemented the C‐SSRS. To our knowledge, this is the first study to assess the impact of switching from a “DTS” question to a standardized and validated suicide screening approach. We found an association between C‐SSRS implementation and a decrease in screening rates (92.8% to 84.6%); a meaningful increase in the proportion of screened patients screening positive (1.5% to 2.2%); and among those screening positive a decrease in the proportion of patients with documentation of psychiatric care within 90 days (64.0% to 52.5%), but an increase in documentation of psychiatric care within 90 days among all patients seen (0.87% to 0.96%). Similar trends were observed for the subgroup of patients with MDD. A subgroup analysis compared patients screening positive using the DTS question to those identified with moderate or high risk by the C‐SSRS, based on the logic that these are the groups warranting immediate psychiatric care, and found a slight increase in psychiatric care (64.0% to 66.4%).

The observed decrease in screening rates may be explained by the additional time required for screening using the C‐SSRS. Additionally, some patients may have been unable or unwilling to respond to the C‐SSRS. This analysis used only records of completed C‐SSRS screening, excluding records with incomplete and partial C‐SSRS data. Future qualitative research is required to understand clinician and patient experience with the C‐SSRS and the specific barriers with integrating it into general population clinical workflows.

Systematic screening for suicide risk is a key ingredient of health system interventions such as the Zero Suicide Initiative (27), and with the C‐SSRS, we observed a meaningful increase in positive screenings. This increase may have several potential explanations. First, the C‐SSRS is a more valid and reliable instrument for identifying individuals with suicide risk compared to the single DTS question, and it can determine suicide risk severity (22, 23). Second, the “DTS” question was focused on immediate risk of self‐harm and could have excluded people with low risk. Comparing people identified with DTS with those identified as moderate or high risk with the C‐SSRS, we observed a change from 1.5% screening positive by DTS to 1.3% screening positive with moderate or high risk by C‐SSRS. Third, the COVID‐19 pandemic may have influenced rates of suicide risk (35). A recent analysis in the same population found that COVID‐19 led to a 19% reduction in patients seen in these hospitals, but an increase in those identified as moderate or high risk. Policies such as “shelter‐in‐place” orders may have discouraged utilization of healthcare services for non‐urgent and low‐acuity issues (36).

Considering the overall increase in the proportion of patients with documented psychiatric care out of all patients seen, the decrease in the proportion of patients with positive screenings with documentation of psychiatric care may be partially explained by the fact that the post‐period positive screening cohort includes people identified with low risk. Measures of psychiatric care were primarily inpatient based and may not be appropriate for those with low risk. One advantage of the C‐SSRS is its suicide risk categorization and consequently the ability to direct people with lower risk to outpatient care. When comparing people screening positive by DTS with those moderate or high risk by C‐SSRS, there is a slight increase in rates of psychiatric care, indicating that follow‐up care was prioritized for those at higher risk.

Limitations

The generalizability of this study may be limited due to its focus on adults in one health system. Ideally the study would have incorporated data on suicide attempts and deaths, but these data were not available. This study was unable to distinguish whether mental health or suicide risk was the primary reason for a patient's hospital visit. This study may overestimate follow‐up psychiatric care as our measure included referrals to behavioral health providers, which may not have resulted in an encounter. This study may also underestimate psychiatric care and diagnoses. Much psychiatric care is difficult to access and provided in private practices (37, 38), so those encounters were not documented in this health system's EHR, however these limitations were present in both pre‐ and post‐period so likely did not influence results. Also documentation may not perfectly reflect practice, some screenings may not have been documented, and some documented screenings may not have been asked verbally to the patient. These constraints with documentation and interoperability of mental health information are a critical challenge to research, clinical care, and population health. Finally, this study compared a “DTS” question which measured potential for self‐harm to a suicide risk questionnaire. Despite different screening goals, self‐harm often leads to increased suicide risk and both populations warrant appropriate psychiatric care (39, 40).

CONCLUSION

These findings present evidence that switching from the unvalidated “DTS” question to using the validated C‐SSRS resulted in successfully identifying more patients at risk for suicide and the appropriate level of care needed. Future research may be necessary to examine the experience of clinical staff and patients with the C‐SSRS and to understand barriers to increasing its use. Standardized suicide screenings, if successfully adopted in healthcare settings nationwide, have the potential to efficiently identify people at risk for suicide, providing an important opportunity for prevention.