The International Mental Health Assessment: Validation of an Efficient Screening Inventory

From the outset, based on the needs of EAP counselors and for feedback to respondents, we expected to include common domains of disorders and difficulties, including depression, anxiety, post-traumatic stress, substance abuse, anger, sleep problems, and interpersonal conflict. A number of well-validated inventories exist to measure these specific problems, an empirical literature that forms the backbone of the development of the IMHA, as described below. But these narrower inventories are not intended for screening in a non-clinical population, and deploying several as a group would require many items, often overlapping in content, and with a cacophony of response options.

Such an approach would also ignore the evidence for the co-morbidity of common disorders and over-emphasize categorial distinctions (Conway et al., 2019; Kotov et al., 2017). Symptoms across dozens of diagnoses have been shown to aggregate into overarching spectra (externalizing, internalizing, psychotic experience; Caspi & Moffitt, 2018; Conway et al., 2019; Kotov et al., 2017) which in turn have been shown to aggregate into a general dimension of psychopathology (Caspi & Moffitt, 2018; Conway et al., 2019). High scores on general liability associate strongly with a family history of psychiatric illness, brain function, developmental history, and life impairment (Caspi & Moffitt, 2018), and with underlying genetic vulnerability for psychopathology (Pettersson et al., 2016; Selzam et al., 2018). Thus, for efficiency and to reflect a growing scientific consensus, a dimensional approach was preferred, with familiar categories nested into broader spectra and a meaningful overall score.

Existing broad inventories, for example the Symptom Checklist and Brief Symptom Inventory (BSI) platform (e.g. Derogatis & Derogatis, 2001) and the Behavior and Symptom Identification Scale (BASIS-32; Eisen et al., 1999), are designed for use in medical settings and clinical intake, not for screening in the workplace. Furthermore, both inventories assess perceptions of difficulty (BASIS) or being bothered by symptoms (SCL), which conflate presence of a symptom with social and cultural norms and expectations, including ideas about how frequent something ought to be. While this approach may be justifiable in a clinical setting, where distress is a key factor, it inhibits the comparison of experiences across individuals or groups on any factor other than ‘perceived distress’, which may stem from specific mental problems, from a lack of social support in the context of normal challenges, or from cultural or role expectations. This is discussed in more detail below.

Inventories aimed at a general population include the Outcome Questionnaire-45.2 (OQ-45; Lambert et al., 2004), a measure of symptoms designed to assess functioning and change during clinical treatment, which has been translated into many languages and is used at clinics around the world. It aims to measure three broad categories (Symptom Distress, Interpersonal Relations, and Social Role Functioning), but no later studies have replicated the intended structure (e.g. Thalmayer, 2015) and the length of 45 items is unnecessarily long for a survey primarily validated for use as a single, total score. The broad Adult Behavior checklist (ABCL; Achenbach & Rescorla, 2003) provides scores on 24 scales within four domains, and has also been translated into many languages, but at 126 items it is problematically long. People vary in their willingness to complete surveys, and shorter measures can have advantages in terms of validity, for example by reducing boredom and fatigue (Burisch, 1984; Goring et al., 2004), increasing response rates and reducing costs (Edwards et al., 2004), potentailly without reducing predictive validity (Kemper et al., 2019; Thalmayer et al., 2011). Both inventories use vague, relative response options. For example, in the ABCL respondents are asked to rate their family relations as worse than average, average, or better than average. But expectations will vary around ‘average’ family relations, and few will be well-informed about typical levels of family conflict or closeness across their society.

Similar to the current project’s goals are two proprietary instruments, the 120-item, 10-subscale, Employee Assistance Program Inventory (EAPI; Anton & Reed, 1994), and the 96-item Spectra Indices of Psychopathology, which provides hierarchical-dimensional assessment (Blais & Sinclair, 2019). Both are intended for counseling intake and are longer than ideal for community and employee samples, and their proprietary nature inhibits collaborative scientific uses. The first was only available on paper, making it unsuitable for online and app-based testing. A shorter candidate is the 38-item Mental Health Inventory (MHI; Veit & Ware, 1983), which provides overall scores on distress and well-being, and five subscales. This inventory has the advantage of brevity, but excludes most content related to ‘externalizing’ problems (e.g. substance abuse, anger, conflict; Conway et al., 2019), and all three inventories use vague response options (e.g. not at all, a little bit, moderately, quite a bit, or completely true).

A common problem across almost all inventories for psychological disorder symptoms is vague response scales. When not focusing narrowly on perceived difficulty, inventories typically ask for subjective, relative frequency: Did you experience this symptom “rarely”, “sometimes”, “frequently”, or “almost always” (OQ-45)? Substantial psychometric work during the rise of survey research in the 20^th century compared relative versus absolute frequency choices (reviewed by Friedman & Amoo, 1999 and Schaeffer, 1991). While many problems with relative frequencies have been demonstrated (described below), assessing absolute occurrence was argued to be more demanding on participants, especially in the case of vague, subjective events (Bradburn & Miles, 1979). It is also probably relevant that survey data is analyzed solely using the numbers assigned to options, and may have come to put less weight on the wording than is warranted. While the numbers on a Likert scale suggest interval scaling, which is generally how analysis proceeds, descriptive terms are not necessarily perceived as equal steps (Brown, 2011; Friedman & Amoo, 1999; Loevinger, 1957; Schriesheim & Novelli, 1989). Despite these issues, the norm for clinical scales to use relative qualifiers has become so strong that consideration of this choice is now generally absent from reports on new inventories.

We find this norm problematic for many reasons. First, there is a lack of precision as to what is being measured – the occurrence of a symptom, or a feeling or judgement about it? Ratings of relative frequency that use vague qualifiers or intensifiers focus comparisons on an internal, implicit standard and are thus shaped by many factors beyond frequency (Schaeffer, 1991). Consider the difference between answering whether you lost your temper last month or whether you lose your temper “often”. The former is a reasonably factual question, referring to an experience that is visible to others as well as oneself, while the latter may feel like a moral judgement: “Often” implies that you lose your temper more than is “normal” or “average”, but compared to whom? Relative-frequency ratings have also been shown to be influenced by expected frequency, with different standards for rare events (earthquakes) versus frequent ones (rain), as well as by valence: Something unpleasant that occurs three times a week will be described with a qualifier indicating more frequency than something pleasant occurring at the same rate (Schaeffer, 1991).

Secondly, there is the difficulty of comparing scores between groups. Relative frequency is shaped by social norms and expectations, leading, for example, to differences among subgroups in how many days of the month are associated with “very often”, “pretty often” and “not too often” (Bradburn & Miles, 1979). Vague options are also vulnerable to response styles, such as acquiesce bias (a general tendency to agree to survey items) or extremeness in responding (sticking to the low and high ends), both of which have been shown to covary with cultural traits (e.g. power distance, individualistic values, uncertainty avoidance; Johnson et al., 2005), which vary between nations, as well as between individuals.

Further complicating cross-group comparisons is the fact that vague response scales are subject to reference group effects (e.g. Heine et al., 2002; Van de Gaer et al., 2012). For visible traits, such as Conscientiousness, people can only self-report tendencies assessed with vague qualifiers by comparing themselves to those they know: classmates, colleagues, friends, family, and acquaintances. They are limited by social circles and self-evaluations are strongly shaped by local expectations. This leads to counterintuitive findings, for example, national mean scores for self-reported Conscientiousness correlate poorly with objective society-level indicators (e.g. accuracy of public clocks, efficiency of postal workers, public-sector corruption; Heine et al., 2008; Oishi & Roth, 2009), and students from the world’s best schools in terms of academic achievement rate themselves lower on academic talent than those from low-performing schools (Shen & Tam, 2008).

This measurement challenge is further complicated in the case of psychological disorder symptoms, which are less visible. How depressed or anxious someone feels, or how well they sleep, is not obvious to an outside observer. Most people will know about such experiences only for intimate friends and family. Thus, what does it mean when an individual answers that they worry “a lot” or “seldom”? Only that this is the true in comparison to a few close others, or in relation to local assumptions, for example media depictions of ‘normal’ functioning.

The problems of vague response scales are avoided in the International Mental Health Assessment (IMHA) by using a response scale referring to specific frequencies within the last month. While testing the cross-cultural applicability of this inventory is beyond the scope of this report on the development of the inventory, this value was taken into consideration from the outset, to maximize validity in the diverse society of the United States, and for comparing across groups, be they cultural, subcultural, regional, or by gender, age, social class, education level, etc. The impact of response biases is also addressed by using items that are as concrete and behavioral as possible (e.g. “I had difficulty falling asleep”, “I argued with friends or family”, “I drank enough to pass out”) to insure state rather than trait assessment (Hopwood et al., 2022). Specific items also minimize the key advantage of relative-frequency ratings, that they reduce respondent effort in the case of hard-to-quantify experiences. Note that concrete specificity does not preclude assessing for distress or impairment, as in: “Not getting enough sleep interfered with my daily activities”, “I had difficulty making decisions” and “My worrying got in the way of doing something I intended to do”. Over the long-term, concrete items and a specific, absolute response scale are intended to facilitate the establishment of cross-cultural measurement invariance to allow for cross-group, cross-cultural comparisons (e.g. Fischer & Karl, 2019), and it creates the potential for meaningful item-level comparisons across individuals or groups even where such measurement invariance cannot be established.

The aim of this project was to follow a multi-step, iterative process to create and validate a comprehensive but maximally efficient inventory of common psychological problems, integrating the strengths of existing domain-specific measures (in terms of item content) and the values of hierarchical (spectra) assessment, and using a concrete response format less vulnerable to reference group effects and thus suited for assessment and comparison across groups. Allowing common cross-syndrome symptoms to serve as overall screeners was a means of keeping the scale as brief as possible, to reduce the stress of completing it, and to increase completion rate (e.g. Deutskens et al., 2004) and thus utility as a screening tool.

Study 1 includes the creation of a preliminary 69-item version and its administration to a large group of employees and a sample of clients presenting for services at the EAP’s counseling office, in order to increase the base rate of complaints and to allow for comparison as an initial assessment of validity. IRT analyses using the Rasch Partial Credit Model (PCM) emphasized improving the inventory to meet the goals of reliable and internally-valid subscales and Total Score, good distinction between domains (reducing subscale intercorrelations), and minimizing length. Study 2 includes the administration of the refined 48- to 59-item version to a second large sample of employees, which was split into two random halves. Analyses in the first split-half focused on refinement to meet the same criteria, with exploration at p-factor, spectra, and subscale levels. The split-half procedure allowed for testing a refined version in the second half; calibration of the final version of the IMHA and its psychometric properties are reported for the full sample. In Study 3, the IMHA subscales were compared to 21 scales for relevant subdomains, belonging to 15 established inventories, to assess convergent and divergent validity.

Only anonymous survey data were made available by Canopy to the study authors. The Committee for the Protection of Human Subjects of the Research Compliance Services of the University of Oregon found this use of existing data, which was collected following strict HIPAA (Health Insurance Portability and Accountability Act) guidelines, to be exempt from the need for full institutional review (2019).

The raw data and scripts for running the analyses are available on the Open Science Framework (https://osf.io/vysj4/). Sufficient information is provided for an independent researcher to reproduce all reported results (Open Data) and all reported methodology (Open Materials).

Materials. The preliminary IMHA was created by identifying common psychological and behavioral health problems among community adult and employee populations: alcohol and drug abuse, anxiety, depression, post-traumatic stress, sleep problems, life stress, work disengagement, anger, interpersonal conflict, and protective factors. Prevalence rates, drawn primarily from the Diagnostic and Statistical Manual of Mental Disorders (DSM–V; American Psychiatric Association, 2013), were an important criteria in choosing domains for inclusion. Other domains (e.g. eating disorders, psychotic experiences, the identity disturbances of borderline personality disorder) were considered but excluded, because these less prevalent conditions can be assumed to share symptoms or be comorbid with more common problems such as anxiety and depression (e.g. Conway et al., 2019; Eaton et al., 2010; Kotov et al., 2017). Allowing common cross-syndrome symptoms to serve as screeners was a way to keep the inventory brief. Although we aimed to make reliable distinctions between domains, and retained them to facilitate psycho-education and communication with clinicians, the subscales were expected from the outset to correlate with each other due to established patterns of covariance among disorder categories (Conway et al., 2019). Spectra for internalizing and externalizing tendencies and a total score were planned to provide assessment of broader domains of psychological difficulties, allowing for three levels of analysis. This hierarchical approach allows clinicians to use familiar terms while raising consciousness about their overlap.

To make best use of the large body of empirical literature on clinical assessment, the strategy for creation of the preliminary IMHA was to seek content convergence among validated inventories for each domain of interest. For each domain, three to five inventories were identified (detailed in Supplemental Table S1). Public domain inventories were favored and viewed at the item level. In some cases, proprietary instruments were viewed, in this case with a focus only on content areas. Inventories were compared to each other, to clinical criteria in the the DSM–V (American Psychiatric Association, 2013), and to empirical studies of the constructs in order to define regularities in key content. For example, for post-traumatic stress five inventories were viewed. At least three included items about 13 symptoms: intrusive memories, reliving, sleep disruptions, irritability, emotional suppression, numbness, dissociation, avoidance, physiological arousal, flooding, concentration difficulties, nightmares, and vigilance. Other symptoms, like memory loss, loss of interest, and pessimism were included on fewer inventories and are less specific to post-traumatic stress. Of the core 13, sleep problems, irritability, and concentration items were already included for other IMHA domains. IMHA items were thus developed for the remaining 10 symptoms, focusing on specific behaviors and feelings and fit to the response scale, following best practices in item writing (e.g. Clark & Watson, 1995). The initial version was intentionally over-comprehensive, including more items than expected to be necessary for the final version (Clark & Watson, 1995).

The response scale was developed to refer to specific frequency in order to minimize reference group effects. Respondents are asked to recall how often a behavior or feeling occurred in the last month, and to answer on a 7-point scale of frequency: daily, half the days, about twice a week, about once a week, about twice a month, monthly, not in the last month. (The choice of these options was also guided by their potential to correspond to an interval scale with a natural log transformation if quantified as days per month, though this was not done in the analyses). The Protective Factors items were reverse-scored for the unidimensional model.

An initial draft inventory was reviewed by clinical psychologists and piloted with individuals who described their perceptions and reactions while or after completing it. While some noted appreciation in seeing a symptom on the scale (e.g. intrusive thoughts) made them feel less alone, in general they reported that the questions were ‘heavy’ and reminded them of the ‘dark’ side of life and they found it important to be as brief as possible. An a priori construct map (in online supplemental materials) details the expected meaning of scores on the Total Score and subscales based on clinical experience and existing surveys in each domain. For the pilot sample, feedback text was desired, so estimated cut-off scores for low, medium, and high risk were rationally developed by the first two authors and EAP psychologists, based on a face-valid understanding of the distress and impairment implied by different frequencies. The initial cut off scores and observed percentage of participants who fell into each category, relevant primarily for the feedback system used by Canopy Wellbeing, are presented in Supplemental Table S2.

Participants. A total of 5,307 individuals completed the preliminary survey, including 5,170 public employees of either a medium-sized west-coast city or a west-coast state, and 137 clients who presented for services at the counseling office of the EAP. Multiple completions by the same person and responses from anyone under age 18 were excluded. Age, gender, and ethnicity information for the two Study 1 samples are reported in the left-hand side of Table 1. For this study, we used all cases available in the existing data, assuming the sample size was more than sufficient for the planned PCM analyses, following De Ayala (2009), who suggests a loose rule of thumb of about 250 for Partial Credit models, though understanding that this is not precise and that formal power analysis is not often done for IRT modeling (Zimmer et al., 2022).

Procedure. The EAP made the pilot version of the survey available free-of-charge to two employers, with the understanding that anonymous information from their employees would be used to test and develop the IMHA and the full product in which it is embedded, which includes a proprietary interface and feedback protocols. Emails from the EAP to employees of these organizations notified them of the opportunity to complete the survey and to receive individualized feedback. A report that appeared after survey completion categorized each respondent in terms of low-, medium-, and high-risk categories for each of 10 domains, and included links to articles about self-care and the phone number for EAP counseling services. Confirmatory answers to three ‘red flag’ items that indicated potentially violent feelings or being a victim of violence led to immediate feedback urging a call to the counseling center. Aggregate, anonymized summaries were delivered to employers, advising them about general areas of concern for their employee population.

Analyses. IRT estimation relied on Rasch family partial credit models (PCM2; Andrich, 1978; Masters, 1982; Rasch 1960/1980). The unidimensional model was tested in R package Test Analysis Modules (TAM; Kiefer et al., 2017) using marginal maximum likelihood estimation; multidimensional models in R package supplementary item response theory (sirt; Robitzsch & Robitzsch, 2020) also with marginal maximum likelihood estimation. For all models we report the number of ‘steps’ (six per item, given the seven-point response scale) with mean square weighted fit outside a ¾ - 4/3 tolerance (Adams et al., 1997; Adams & Khoo, 1996). Following those authors, our a priori standard was to avoid more than 5% misfit, as this would indicate that too many items correspond poorly to others in the set. Also assessed were PCM and Cronbach alpha reliability, and scale intercorrelations. Raw mean scores and PCM scaled scores (thetas) were compared between the employee and clinical samples. Models were compared using difference in deviance (χ²), AIC and BIC (Schwarz, 1978). ‘Wright Map’ graphical representations (Wilson, 2005) were used to view the score distribution on the latent trait for individual items, to help identify redundant items and to aid scale refinement.

Descriptive statistics. In the full sample, all options (0 - 6) were used for all items with two exceptions. The item “My partner physically hurt me” had no responses beyond “monthly” (1). The item “I got so angry I had a physical altercation with someone” was not responded to above “about twice a week” (4). The six protective factors items had the highest mean scores, from 3.3 to 4.7. Raw means for the other items ranged from 0.01 to 3.08 (reported in Supplemental Table S2.)

PCM Estimation and Exploration for Refinement. The unidimensional Rasch model had high reliability (.95; personal separation .93; Cronbach alpha .96). However, 11% of the 413 item-steps had mean square weighted fit outside the recommend tolerance, indicating lack of correspondence with other items in the inventory. The problematic steps were for three substance-use items: one (“I had 3 or more alcoholic beverages in a day”) may not relate closely to psychological disorder symptoms; two were very infrequently endorsed. There was also problematic fit for five of the six protective factor items.

Model fit was better for a 10-dimension Rasch model, χ² (63) = 27,244, p < .01, person separation reliability (details reported in Supplemental Table S3). Only five items (< 5%) had steps outside the acceptable range. Reliability and intercorrelations among the scales are reported in Table 2. Two scales had low IRT reliabilities, and three low Cronbach alpha. Not surprisingly, the internalizing subscales (Depression, Anxiety, Post-Traumatic Stress) correlated highly with each other (.90-.93). Mean thetas (PCM scaled scores) for the employee and clinical samples are graphically contrasted in Figure 1. The clinical sample had significantly higher scores on all scales, as expected.

Wright Map graphical representations of the score distribution on the latent trait were viewed for each subscale to aid scale refinement. These are included in Supplemental Figure S2 with detailed explanation and examples. While the histograms show normal distributions of the latent traits, the distinctions between steps are most reliable at the higher end. Given that this is a screener and that less than a tenth of the data came from a clinical population, this was seen as appropriate: most people from the community will have relatively low scores, and it is not efficient to make fine-grained distinctions between non-problematic score levels, e.g. between extremely low and very low depression scores. Instead, distinctions are most relevant at the higher end. These charts are also useful for identifying items that make the same distinctions and thus may be redundant. For example for Depression, four items, while covering different content, were seen to make the same distinctions in this sample, adding little incremental value.

In Study 1, a preliminary 69-item version of the IMHA was developed to meet the goals of broad assessment of common psychological problems for use in a normal adult population, with concrete behavioral items and an objective response scale. It was administered to a large employee sample and a sample of counseling clients. PCM analyses were used to identify potential modifications to increase reliability and decrease scale intercorrelations and length.

The Total Score was seen to have good reliability: the items worked together to define overall psychological functioning, with the main exception of the Protective Factor items, which did not distinguish well between the trait levels defined by other items. This result was not surprising, given that this subscale included the inventory’s only reverse-scored items, and that they were drawn from domains of literature (positive psychology, personality, values) outside clinical psychology. This subscale was included so that feedback could cover a positive domain and to lighten the experience of completing the survey. On balance, however, given the items’ lack of correspondence with the IMHA’s core content, keeping the assessment brief was deemed to be a higher priority, and this subscale was dropped.

For clinical reasons, we retained screener items (suicidal thoughts, partner violence) and the other nine subscales. Of these, the six subscales of the internalizing spectra had good reliabilities but relatively high intercorrelations. We sought to maximize distinctions, while accepting inevitable intercorrelation. For these scales, 23 items were removed based on (a) Wright Map indication of redundant coverage, and (b) in PCA, not loading highest on intended subscale, where removal did not reduce match to intended structure.

Three subscales, Substance Abuse, Anger, and Partner Conflict, had low reliabilities. Substance Abuse (EAP reliability .65; α =.61) was more categorical and less normally distributed than the other subscales, with relatively few people reporting problems. As the subscale appeared to conflate occasional social drinking with dependency, we separated it into use and abuse subcomponents, and added items about other drugs with the aim of increasing reliability. Likewise, Anger focused on quite disruptive behavior, which led to low endorsement and reliability (EAP reliability .77; α =.34). We addressed this by removing the most extreme item and adding an ‘easier’ item about feeling rather than expressing anger. For partner conflict (EAP reliability .65; α =.61), we added items to assess for interpersonal conflict beyond romantic relationships and added an item about being threatened by a partner to increase coverage and reliability, including the ability to screen for dangerous situations.

Materials. The changes described above led to a refined 48-59 item version of the IMHA. The number of items varied by participant because for three subscales, Interpersonal conflict (3-8 items), Substance Use (4-6), Substance Abuse (0-6) some items were only shown based on responses to prior items. This approach reduced length to streamline responding, while maintaining coverage where it would be most useful. Additionally, three outcome variables were administered: Participants were asked if they were currently in treatment with a mental health professional; to rate their overall work performance on a scale of 1 to 10 for the last month; and to estimate the number hours of work missed due to personal concerns for the last month.

Participants & Procedure. A total of 4,048 individuals completed the survey on a proprietary smartphone app. These were employees and spouses from over two dozen medium-sized employers (with 50 to 500 employees) in the United States, predominantly located in the Pacific Northwest, contracting with Cascade Centers’ for EAP services. Feedback was generated and delivered as described for Study 1 using updated protocols. Multiple completions by the same person and responses from anyone under age 18 were excluded from the data delivered to the authors of this report. Age, gender, and ethnicity are reported in Table 1. Because the app required answers to move through the survey and only completed surveys were delivered, there was no missing data. Eight substance abuse items were only asked if the participant endorsed any use, and empty answers were coded as zero. The 831 respondents who said that they were not in a romantic relationship were not asked five items specific to partner conflict.

Analyses. Analyses proceeded as for Study 1 in the first split half of the data, allowing for the possibility of dropping items and testing an interim version in the second half. Here instead of a Total Score, spectra and super-spectra were used to better match scoring to theory. A three-spectra model was created based on typical patterns of comorbidity reported by Conway and colleagues (2019). Depression, Anxiety, and Post-Traumatic Stress were included on an Internalizing spectrum, Substance Use and Abuse and Anger on Externalizing, and the remaining scales (Sleep Problems, Life Stress, Interpersonal and Partner Conflict, Work Disengagement) on a Life Difficulties spectrum. P-factor included Internalizing and Externalizing but not Life Difficulties. The full sample was used for final calibration. Confirmatory factor analysis was performed on the final hierarchical model, with P-factor, spectra, and the associated five subscales. For the outcome questions, regression analyses were used to test associations with each scale, after taking into account the other subscales, age and gender. Logistic regression was used for the binary item about seeking mental health treatment. Poisson regression was used for work hours missed, as the skewed responses included many zeros.

Descriptive statistics. To improve on the rationally derived cut-off scores used for feedback during piloting, we provide ‘norms’ in the form of percentiles in Table 3. Because of the skewed sample in favor of women and gender differences in scores (see below) these norms are separated by gender. The proportion of the sample who scored zero (no symptoms in the last month) on each scale and scores associated with the 50^th, 75^th, 90^th and 95^th percentiles are shown: i.e., the score that distinguishes the less symptomatic half of the sample from the half with more symptoms (50%), the score that indicates being in the lower three quarters of the sample (75%), and the lower 90% and 95%. Scores for an individual or a group from a similar population (e.g. a North American employee) can be compared to these norms. For example, a woman with a depression score of 27 could be understood to be in the top 10% for these symptoms, with a score higher than 90% of her peers.

In the full sample, raw item means ranged from 0.01 to 2.85. All options (0-6) were used for all items with one exception: “Family or friends suggested I should cut down on my drinking or drug use” had no responses in the “about twice a week” range. Six other items had one or more response options with five or fewer cases. For PCM analysis, responses were adjusted downward to create a minimum of five cases per cell, with the top response category left empty.

Partial Credit Model Estimation and Exploration for Refinement. In the first random half of the data, the P-factor had EAP reliability of .90 and 16 steps (7%) with fit outside 3/4 - 4/3 tolerances. All misfitting steps were from the five substance use items about using a specific intoxicant. Only the use item “I drank enough to feel intoxicated” had no misfitting steps. Indeed, the use of substances, while a pre-requisite for abuse, does not itself constitute a mental health problem, and we thus removed five items, retaining only that about intoxication. This 35-item P-factor in the second random half of the data had EAP reliability of .90 and only 4% of items with fit outside tolerances. Reliabilities for the three-spectra model using 54-items in the second random half were .93, .72, and .91, for Internalizing, Externalizing, and Life Difficulties, respectively, with no misfitting steps. The three spectra were highly intercorrelated (r_IE = .72, r_IL = .91, r_EL = .69), which is unsurprising given their shared fit to a total score.

Exploration of the multidimensional model started with all 59 items in the first half of the data with 11 maximally-disaggregated subscales, separating Interpersonal from Partner Conflict and Substance Use from Abuse. This model had less than 5% misfitting steps, but, not unexpectedly, the two conflict scales (EAP reliability .78 and .81) were correlated .90, and these were thus seen as better combined. Substance Use and Abuse (EAP reliabilities both .67) were correlated .88, though item correlations for use indicated only small associations between alcohol and drug items. Based on their lack of fit to the P-factor either empirically or conceptually, the five use items were dropped going forward.

Depression, Anxiety, and Post-Traumatic Stress correlated at or over .90 with each other. Principal components analysis and perusal of Wrights Maps (shown for the full data set in Supplemental Materials Figure S2) were used to search for items not loading on the intended scale or contributing additional coverage to the subscale, respectively. However, reliabilities under the IRT model were not improved by the removal of two candidates that were identified (“It was hard to control my worrying” and “I felt jumpy or easily startled”) and intercorrelations were reduced only slightly. Given the acceptable length for practical purposes and the goal to adapt the inventory to other languages and contexts, where a larger item pool could have advantages, it was decided to retain the items. Anger was also explored, as its reliability was low (EAP .71). An exploration of interitem correlations suggested no candidates for removal that could improve Cronbach Alpha, and removing the two least-correlated items did not improve IRT reliability.

Fifty-four items were thus retained for the final version of the IMHA. The three-level hierarchical model is displayed graphically in Figure 2. Reliability, IRT correlations and mean thetas (PCM scaled scores) for men and women for the nine-subscale and three-spectra models in the full dataset are reported in Table 4. These indicate that women had higher scores on every domain except Substance Abuse. Sex differences are displayed graphically in Figure 3. Model fit in the full dataset for the three-spectra, and nine-subscale models, also compared to a unidimensional total score including all 54-items, are reported in Table 5. According to χ², AIC and BIC, the more elaborated models fit significantly better than less elaborated models, with best fit for nine-subscales. CFA fit for the formal hierarchical model, excluding Life Difficulties, was strong, χ²(3) = 41.01, CFI = .996, TLI = .987, RMSEA = .056, SRMR = .011; standardized loadings are shown in Figure 4.

Outcome variables. Higher self-rated work performance over the last month (range = 1 - 10; M = 8.1; SD = 1.3), was significantly predicted (p < .001) by being older (β = .08), by lower scores on Depression (β = -.32), Sleep Problems (β = -.12), and Work Disengagement (β = -.23), and, less intuitively, by higher scores on Post-Traumatic-Stress (β = .09). Together age, gender, and IMHA subscales accounted for 29% of variance in this rating. Hours of work missed (not a required response) was estimated by 57% of the sample (n = 2,322), with answers ranging from 0 (35% of respondents) to the maximum possible of 180 (M = 4; SD = 10.4). It was significantly predicted only by Life Stress and Work Disengagement (p < .001). Seventeen percent of the sample (n = 695) responded that they were in treatment with a mental health professional; this was significantly predicted by higher scores on Depression, Post-Traumatic Stress, and Work Disengagement (p < .001). The full model accounted for 15% of variance in treatment seeking.

In this study, a 59-item version of the IMHA, refined based on the results of Study 1, was administered to a large employee sample. Good fit and reliability were found for a 54-item version, including a 35-item hierarchical model of P-factor, Internalizing and Externalizing spectra, and their associated five subscales: Anxiety, Depression, and Post-Traumatic Stress; and Substance Abuse and Anger. The full IMHA also includes a 19-item spectra of Life Difficulties, with subscales of Life Stress, Sleep Problems, Workplace Disengagement, and Interpersonal Conflict. These items and subscales are not conceptually solely related to mental health and thus are not included on p, although the scales and the spectrum are strongly associated with the other spectra, especially Internalizing. These complaints about sleep length and quality, recent bad experiences and/or time deficits, conflict with close others (with most items focused on being a victim in the conflict), and missing work due to personal problems, could be consistent with internalizing or externalizing tendencies, but also with specific or temporary life circumstances such as the birth of a child, the death of a loved one, or social or economic hardship. Those with high scores may benefit from counseling or wellness interventions, without their scores indicating a tendency toward a disorder. We also find it advantageous to retain these subscales and spectra to help put mental health and functioning into a broader context of a person’s life. It will also be useful to determine if the same strong associations are seen in future cross-cultural studies, for example in poorer countries where the incidence of Life Difficulties is higher.

The best fit was seen for the nine-subscale model. The three internalizing scales, Depression, Anxiety and Post-Traumatic Stress, remain highly intercorrelated (.90 to .94), which is a limitation of that model and indicates the advantage of the spectra model. However, intercorrelations are lower for the Externalizing scales (.51), and separate scores may be useful in some contexts, as long as the high intercorrelations among the internalizing scales are acknowledged. The three nested models otherwise have strong measurement properties, and each may be the best level of analysis depending on the intervention, tracking, or research goal.

Materials. The 54-item version of the IMHA used in Study 2, as described above, was administered with 12 other inventories, detailed in Table 6.

Participants. Five hundred eight community adults recruited through Prolific completed the 13 surveys through a Lime Survey interface. Eight cases were rejected due to failure of more than half of eight attention checks, which was sometimes combined with very short completion times (under 5 minutes for 170 items), leading to a final sample of 500. This was estimated to be a larger sample than needed for the planned correlational analyses, with moderate to high anticipated correlations (Cohen, 1992). Participation was restricted to individuals currently residing in the United States who speak English fluently to maximize comparability with Study 1 and 2 samples. Age, gender, and ethnicity are reported in the far right-hand side of Table 1.

Procedure. Institutional ethical review of this study was not available at the lead authors’ European institution for a survey study. However, survey responses were entirely anonymous and data collection complied with the ethical standards of the American Psychological Association. A debriefing page provided contact information for free phone and sms crises lines. A payment was made based on an hourly rate of US$10 per hour. The surveys were administered in five possible orders, and item presentation within most surveys was randomized. The format required answers to move through the survey, thus, there was virtually no missing data, with the following exceptions: Errors at the start of collection meant that some items were not presented to the first eight participants. The first WOS scale used a ‘write in’ format for number of hours, and responses for seven participants who wrote “not applicable” or described an issue that made them miss work were removed. Participants who reported that they were not in a relationship responded, in a few cases, to partner conflict questions, which were excluded from analysis.

Analyses. Pearson correlations were calculated for all subscales and inventories. A correlation of .70 or higher was seen as indicating that the two inventories measure highly overlapping content (convergent validity), as it indicates that half their variance is shared. It was hypothesized that the established inventories would have high correlations, and their highest correlations, with their related IMHA subscale. Thus, a correlation of this value with an intended scale was interpreted as providing evidence of convergent validity. Otherwise, correlations are interpreted following Cohen’s (1988) conventions: .10 or lower as weak; .30 as moderate; .50 or larger as strong.

Scale scores and psychometric properties are reported in Table 7, along with correlations between the IMHA subscales with each other and with the 21 external scales, belonging to 11 inventories. Four IMHA subscales (Depression, Anxiety, Post-Traumatic Stress, Sleep) were each compared to a single other inventory. In these cases, all anticipated correlations were .70 or higher and the intended inventory had its highest correlation with the intended IMHA subscale.

The other five IMHA scales were compared to inventories with more than one subscale or to more than one inventory, which allows for further distinctions to be explored. IMHA Life Stress was compared to the 10-item SOS and its two subscales: Personal Vulnerability and Event Overload. While both the SOS Total and Event Overload had correlations over .70 with IMHA Stress, only Event Overload had its highest correlation with IMHA Life Stress. The 2-item IMHA Work Disengagement subscale was not associated above .62 with any WOS subscale; these were also not strongly associated with any other IMHA subscales. IMHA Partner Conflict (separated here from Interpersonal Confict which also includes three non-partner items) associated .70 with the Tool for Intimate Partner Violence Screening (HITS), but not above threshold with the Couple Satisfaction Index, which was not strongly associated with any IMHA subscale. IMHA Substance Abuse was more associated with AUDIT (alcohol abuse) than DAST (drug abuse). IMHA Anger was not strongly associated with any BPAQ subscales, which were also not strongly associated with any other IMHA subscales.

In Study 3 the subscales of the IMHA were compared to 11 other inventories with 21 unique subscales to assess their convergent and divergent validity in an online sample of adults residing in the United States. Observed associations supported the convergent validity of Depression, Anxiety, Post-Traumatic Stress, and Sleep subscales, based on strong associations with a key inventory. The other five IMHA scales were compared to inventories with more than one subscale, or to more than one inventory, allowing for examination of specific distinctions.

In three cases, logical results confirm the focus and/or format of IMHA subscales. IMHA Life Stress primarily overlapped with SOS Event Overload, which was designed to assess ‘impinging demands’ as opposed to ‘depleted resources’ with the Personal Vulnerability scale. The latter includes internalizing content (underlined by associations with IMHA Depression and Anxiety) that was intentionally excluded in the IMHA subscale to maximize scale separation. IMHA Partner Conflict associated strongly with HITS (partner violence), but not with the Couple Satisfaction Index (CSI). This result was not surprising given the positive framing of CSI items, and the observation in Study 1 of low associations between IMHA symptoms and reverse-scored Protective Factors items. These do not appear to form the ends of a single pole, but to assess different aspects of the relationship experience.

IMHA Substance Abuse was more associated with AUDIT alcohol abuse (r = .78), than DAST drug abuse (.49). One explanation for this difference may be methodological - the AUDIT and IMHA use very similar response options, whereas DAST has a yes/no format. This supposition is supported by correlations below .50 for nearly synonymous items: e.g., IMHA “I felt guilt or remorse after drinking or using drugs” and DAST “Do you ever feel bad or guilty about your drug use?”. Different dynamics between alcohol and drug use may also be at play. For efficiency, the IMHA scale integrates alcohol and drug problems (of seven items, two are specific to alcohol, and five refer to either/both). However, internal consistency is high (α = .92), and we find integration suitable for screening purposes.

Two IMHA scales did not have correlations .70 or higher with the intended inventories. The strongest association for IMHA Work Disengagement was with WOS Presenteeism (r = .58). This is logical given the two IMHA items are about distraction and missing small amounts of work. While not meeting our cut-off, we found this acceptable for this two-item scale, intended to provide a minimal overview assessment of workplace impact.

IMHA Anger, focused on feeling and/or expressing anger and/or arguing in a variety of contexts, had the expected convergent patterns with the associated BPAQ subscales, but the magnitudes were only moderate. There may be a methodological explanation: BPAQ items are answered on a scale of “extremely characteristic (versus uncharacteristic) of me” drawing on schemas about the self rather than on the concrete occurrence of a behavior. This hypothesis is supported by a correlation between nearly identical items: BPAQ “I have become so mad that I have broken things” and IMHA “I got so angry I broke something” of only .32. Future studies could further explore this subscale’s associations. For the time being, given the lack of anger inventories in the public domain (we found no others) we stand by the value of the IMHA, as the length of the BPAQ and its wordy, complex items and response options make it less suitable for workplace screening or cross-cultural research.

A limitation here is the lack of divergent validity between the internalizing scales. PHQ Depression also correlated .70 or higher with IMHA Anxiety and Sleep Problems; GAD Anxiety with IMHA Depression and Post-Traumatic Stress; and PCLC Trauma, SOS Total and SOS Personal Vulnerability with IMHA Depression. To assess for which items impacted divergent validity, a post hoc exploration of correlations between IMHA Internalizing Spectra items plus Sleep Problems and the four external scales identified two IMHA items: “I felt hopeless about the future” (Depression) and “It was hard to control my worrying” (Anxiety), both of which had highest correlations as intended, but additional correlations > .70 with other external scales. These two items are valid for assessing the internalizing spectrum, but appear to provide less subscale discrimination. Note that this problem is not unique to the IMHA. Intercorrelations among the external scales in Table 7 are generally even higher, for example .84 between GAD and PHQ. These results underline the relevance of spectra assessment.

This report describes the development of an efficient inventory for screening of psychological disorder symptoms in a normal adult population at three levels of analysis. It was intended to be as brief as possible to encourage completion and minimize distress, with subscales for multiple common problems to allow for feedback and communication with counselors, and a hierarchical structure including spectra and p-factor to account for the consistent overlap between traditional domains. Both specificity and the potential for intra- and inter-individual, as well as cross-group and cross-cultural comparisons were maximized by reference to specific behaviors and a response scale based on absolute frequencies of days in the last month. Item development drew on a large base of existing literature, using systematic comparison of the core content of common categories of disorders to create an integrated inventory. The intended uses of the IMHA are screening for prevention and early intervention, tracking individual change, and cross-group and cross-cultural research.

An initial 69-item version with 10 subscales was tested in a large employee sample and a sample of counseling clients using IRT (PCM). Refinements to increase subscale reliabilities and decrease item redundancy and subscale intercorrelations led to a 59-item version, which was administered to a second large employee sample. Tests in the first split half of the data allowed for testing refinements in the second half. At this stage, five substance use items were dropped, as they contributed neither conceptually nor empirically to P or to the Externalizing spectrum. Final calibration and calculation of percentiles for norms is reported using the full sample for three telescoping models: a unidimensional P-factor; three-spectra including Internalizing, Externalizing, and Life Difficulties; and nine-subscales. This dimensional approach situates the inventory in current empirical knowledge (e.g. Conway et al., 2019; Kotov et al., 2017) and increases the utility of the measure. For example, subscales provide familiar information for targeted interventions and tracking, to facilitate communication and access to psycho-education resources. P-factor and spectra can provide a useful overview for an individual or a counselor to track improvement or decline across a range of factors, including for those who do not meet criteria for a disorder but with personality traits that predilect them to internalizing or externalizing tendencies. Alongside this, high scores on Life Difficulties may put other elevated scores into context, or even in the absence of other high scores, indicate the need for support during a challenge or transition such as a new child, grief, unemployment, divorce, or conflict.

The brevity of the IMHA compared to other comprehensive inventories is central to its intended use for screening for early intervention among community adults, not for fine-grained diagnoses (e.g. Kemper et al., 2019). At 54 items, it requires half the time or less than the SCL, ABCL, EAPI, or Spectra. A period in an individual’s life when psychological problems and/or life difficulties are increasing, may also be a time when it is difficult to proactively seek assistance. Drinking, anxious feelings, low moods, poor sleep and concentration, or partner conflict may have worsened over time, leading to habituation. Offering a screener to employees or community groups can help catch problems before a crisis, ideally, increasing awareness and ability to articulate current needs, and thus to access and benefit from resources.

Importantly, in contrast to many popular clinical measures, the IMHA is freely available for scientific and educational uses. The inventory is optimized for research given its brevity, making it practical to include with other variables. The specificity of items (e.g. as recommend by Hopwood et al., 2022) and a concrete rating scale in terms of how many times the symptom occurred, adds to its value as a research tool, serving to diminish reference group effects. These elements also allow for meaningful cross-group comparison of items: the average frequency of a symptom or experience can be compared by treating each item as its own variable (as in Saucier et al., 2015; Westen & Shedler, 2007), to assess for differences even without scale invariance.

This report on the development of the IMHA was limited by not having concurrent data on the presence or absence of diagnoses of disorders. The percentiles from a large sample of indicate the prevalence of such symptoms in this population. Future work could relate scores on the IMHA to diagnoses by structured interview, or those estimated by longer surveys. The studies were also limited by gender imbalance: in Studies 1 and 2 the samples were around 70% women, potentially skewing results. However, this is mitigated by large samples that included over a thousand men, and reporting observed scores and norms separately.

A possible criticism of the IMHA is that it aims to ‘have it both ways’, using traditional disorder categories to suit clinicians, while also integrating evidence of higher-order spectra that are used to avoid reifying arbitrary distinctions. We believe, however, that the presence of traditional categories alongside the spectra serves to raise consciousness, allowing clinicians to integrate what is familiar into this broader, empirically-based context. The continuous measurement approach of the IMHA accurately reflects the continuous nature of psychopathology and its lack of discrete categorical entities (Kotov et al., 2017).

While the IMHA was designed with cross-cultural comparisons in mind, the current study relied solely on a North American sample and this potential has yet to be tested. Future studies should test the IMHA’s utility and norms in other societies, and assess its measurement invariance across languages and groups (e.g. Fischer & Karl, 2019).

Future work should assess the effectiveness and the impact of the response option design The magnitude of difference between the response options, in terms of measurement and clinical practice, should be quantified. Qualitative methods could be used to explore the experiences of participants with regard to the response scale. Estimating the specific frequency of a symptom over the last month is likely more cognitively demanding than making a less precise estimate of “sometimes” versus “often”. However, it also provides for more comparable data, both within individuals over time and between individuals and groups. It would be useful to compare the time spent to complete an inventory in each of the two modes, and to elicit participant reactions to determine how these variations are perceived. It would also be useful to compare responses between the two types of inventories, to help elucidate a ‘folk’ understanding of how many days in the last month constitutes “a lot” or “often” and how this varies across societies, subcultural groups, or age cohorts. Comparisons to other-reports, to clinician assessment, and to diagnoses could be used to compare the validity of the two approaches.

The International Mental Health Assessment (IMHA), an efficient 54-item measure of psychological disorder symptoms is presented. It is made available freely for educational, non-profit or research purposes. This inventory was designed with two goals in mind: to effectively screen and assess degree of psychological problems among adults in the community, and to facilitate research with cross-cultural and cross-group comparisons. It uses concrete items and a response scale based on specific frequencies of symptoms in the last month, designed to minimize reference group effects by avoiding vague comparisons with others. Good reliability for the inventory was established at three levels: P-factor global assessment; three broad spectra of Internalizing, Externalizing, and Life Difficulties; and nine-subscales: Substance Abuse, Anxiety, Depression, Post-Traumatic Stress, Life Stress, Sleep Problems, Anger, Workplace Disengagement, and Interpersonal Conflict.

• Contributed to conception and design: AGT, JM

• Contributed to acquisition of data: AGT, JM

• Contributed to analysis and interpretation of data: AGT, KS, JM

• Drafted and/or revised the article: AGT, KS, JM

• Approved the submitted version for publication: AGT, JM, KS

Julie Marshall is the Chief Operating Officer at Canopy Wellbeing, which sells a product that incorporates the inventory presented, and as such she receives a salary and is part owner of the company. Amber Gayle Thalmayer received compensation from Canopy Wellbeing as an independent consultant while developing the inventory. She completed the manuscript while being supported by Swiss National Science Foundation fellowship PCEFP1_194552.

Requests to use the inventory commercially can be directed to Julie Marshall [email protected] or Anthony Brown [email protected]. Requests to use the inventory free of charge for educational and research purposes can be made at: https://www.psychology.uzh.ch/en/areas/sob/psyges/research/imha.html.

The raw data and scripts for running the analyses are available on the Open Science Framework (https://osf.io/vysj4/).