Assessing the Unidimensionality of Clayton’s Environmental Identity Scale Using Confirmatory Factor Analysis (CFA) and Bifactor Exploratory Structural Equation Modeling (bifactor-ESEM)

For this study, we merged two independent samples of adult students from a Portuguese university. The first sample (n = 174) had participated in a study about contact with nature and environmental attitudes. The second sample (n = 288) had participated in a study about environmental attitudes and well-being. These samples did not differ significantly in terms of age (t = .227, p = .820) or gender (χ² = .900, p = .343).

From this merged sample, we excluded four participants for having over 85% missing data for either of the primary study measures, resulting in a final sample of 458. Most of the sample were women (81%) and most were aged between 20 (1^st quartile) and 30 (3^rd quartile; M = 26.7, SD = 8.92), with a smaller number of individuals older than 30 (see Figure 1).

The two samples of participants completed different batteries of questionnaires. Both sets of questionnaires had a pen-and-paper format. A list of all measures included in these batteries is available on the paper’s project page (available at: https://osf.io/ymb2v/). Both of these distinct batteries included the two measures included in the present study (see Measures section below). All participants completed the study measures individually while in class. This process was supervised by the class teacher. Class groups ranged from between 15 to 25 individuals. Participants did not receive compensation for their involvement.

The Environmental Identity (EID) Scale. Participants in both studies completed the 24-item Environmental Identity (EID) scale (Clayton, 2003), translated into European Portuguese (see Appendix for Portuguese and English versions of the items). Forward-translations from English to Portuguese were performed by the authors, two of whom are Portuguese and fluent in both languages. The items of the scale were presented in the form of a statement (example item: “I think of myself as a part of nature, not separate from it”) to which participants indicated their agreement on a 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree).

The Connectedness with Nature Scale (CNS). For the purpose of establishing convergent validity (the extent to which a scale correlates with another scale purportedly measuring the same construct), participants also completed a Portuguese translation of the 14-item CNS (Mayer & Frantz, 2004). This instrument measures individuals’ affective, experiential connections to nature (example item: “I often feel a sense of oneness with the natural world around me”), and is thus conceptually similar to the EID scale (Tam, 2013). Indeed, as is evident in the example items given, the EID scale and CNS have several items with very close overlap. The items of the CNS are scored on a Likert scale from 1 (strongly disagree) to 5 (strongly agree). In the study sample, the CNS was reliable, omega = .85.

We conducted analyses using the statistical environment R (version 3.6.1; R Core Team, 2019). Raw participant data are available on the paper’s project page (available at: https://osf.io/ymb2v/).

Because we aimed to identify which of several theoretically plausible and empirically identified factor structures best represented our data, we used a confirmatory (CFA) rather than exploratory (EFA) approach to factor analysis. We specifically tested four a priori models (see Figure 2). The first two models were based on the four factors identified by Olivos and Aragonés (2011). To be consistent with the structure identified by Olivos and Aragonés, for these multidimensional models we omitted item 8. Model 1 was a first-order model where the four factors (environmental identity, enjoying nature, appreciation of nature, and environmentalism) correspond to distinct-yet-correlated constructs. Model 2 was a second-order model where a global EID factor accounts for the relationship between the first-order factors. Model 3 tested the basic unidimensional structure suggested by Clayton (2003) and supported by Chew (2019). Model 4 was a bifactor model with one general environmental identity factor and four completely orthogonal specific factors reflecting the subsets of items for each of the four factors of Models 1 and 2. Because specific factors are not mathematically or conceptually the same as first-order factors, we do not assign the same labels. Both Model 3 and Model 4 included 24 EID scale items. All CFAs used the robust maximum likelihood estimator (MLR). The fits of these models were assessed using the goodness-of-fit indices and thresholds suggested by Hu and Bentler (1999; CFI ≥ .95, RMSEA < .06, and SRMR < .08) as well as the comparative measures of AIC, BIC and SABIC. When comparing multiple models, the model with the lowest values for these indices has the best fit.

Next, we used an Exploratory Structural Equation Modeling (ESEM) approach to test whether an alternative factor structure represents the data better than the four a priori models. This analysis was ancillary to our original analytical plan of performing CFAs. ESEM simultaneously addresses the restrictive nature of CFA (which presupposes a simple structure without cross-loadings and thus often inflates factor correlations) and lack of features for model confirmation in EFA (Marsh et al., 2014; Morin et al., 2013). Studies are beginning to suggest that CFA models may be too restrictive to provide acceptable fit for many psychological measures (Marsh et al., 2014). Because we were primarily interested in assessing Clayton’s claim of a single underlying construct, we sought to test a bifactor-ESEM model. For this analysis, we used all 24 EID scale items. First, the optimal number of factors to extract was guided by an initial exploratory factor analysis (EFA) (omega function of psych) with Schmid-Leiman transformation. This analysis supported three specific factors alongside the general factor. Next, guided by the procedure outlined in Fischer and Karl (2019), we performed a separate EFA using the bifactor rotation of Jennrich and Bentler (2011) (fa function of psych) to test a four-factor solution (one general factor, three specific factors). Finally, the loading matrix from this EFA was transformed into structural equations and analyzed using CFA (cfa function of lavaan). In this bifactor-ESEM, items simultaneously load on the general factor and a corresponding specific factor, but unlike for bifactor CFA cross-loadings are freely estimated between specific factors.

To evaluate the interpretability of a mean EID scale score (i.e. mean across all EID scale items) and the degree to which this score reflects a single construct, we calculated Omega Hierarchical (ω_H) to evaluate the proportion of variance in mean EID scale scores due to the general factor. ω_H values greater than .75 suggest they are interpretable as a measure of a single construct despite observed multidimensionality (Reise et al., 2013). We also calculated explained common variance (ECV; Reise et al., 2010). ECV estimates the percentage of common variance due to the general factor. Values for ECV greater than .70 indicate that the factor loadings from a unidimensional model are a good approximation of factor loadings on the general factor in a bifactor model (Rodriguez et al., 2016).

Finally, we examined the consequences of adopting alternative models in terms of their relationship with CNS scores. Specifically, we used Structural Equation Modelling (SEM) to estimate the factor correlations between connectedness with nature and (a) the unidimensional model, and (b) the ESEM-bifactor model. In both models, connectedness with nature was modeled as a latent factor by using a single composite indicator and fixing its error variance. Assuming the unidimensionality of connectedness with nature was appropriate based on an Omega hierarchical value of .74 for the CNS. Following the procedure used by Brown Yost & Finney (2018), the unstandardized error variance of the CNS composite was fixed to .04 based on the equation (1 – r_xx) × var(x), where r_xx was the Cronbach’s coefficient alpha for the CNS (.81) and var(x) was the variance of the CNS composite score (.23). The latent factor representing connectedness with nature was allowed to freely correlate with all factors in each EID model (including general and specific factors in the ESEM-bifactor model).

EID scale inter-item correlations are shown in Table 1. Descriptive statistics for the EID scale items are presented in Table 2. To summarize, most items exhibited low levels of skew and kurtosis (< |1|), suggesting there were no major issues of non-normality. Note, however, that there was a tendency for mild negative skew. Nonetheless, the mean scores for each item were around the middle response option, which suggests results were probably not influenced by floor or ceiling effects. EID scale descriptive statistics are shown in the left-hand panel of Figure 3.

A review of the model fit indices presented in Table 3 indicated that none of the a priori solutions had satisfactory fit considering the pre-defined cut-offs (Hu & Bentler, 1999). Moreover, while the values for CFI, RMSEA, and SRMR generally supported the bifactor model (Model 4) as the best fitting solution, the values for AIC, BIC and SABIC suggested that the first-order model (Model 1) was superior. However, an inspection of the factor loadings and error variances for the bifactor model (see Supplementary Table 1) showed that this model was inadmissible due to a negative error variance for item 23. Moreover, the non-significant loadings for three of the four specific factors indicated that they were redundant in the model. Additionally, the factor correlations observed in the first-order model (M correlation = .89, SD = .05) indicated issues of discriminant validity.

Because none of the a priori models were fully supported, we used an ESEM approach to test whether an alternative factor structure better represents the data. Because we were primarily interested in assessing Clayton’s claim of a single underlying construct, we sought to test a bifactor-ESEM model. The factor loadings of the bifactor-ESEM model are shown in Table 4. The fit of the bifactor-ESEM model was notably better than the a priori models, and indicated good fit for all indices: CFI = .978, RMSEA = .026 [.018, .032], SRMR = .046.

To evaluate the interpretability of a mean EID scale score (i.e. mean across all EID scale items) and the degree to which this score reflects a single construct, we first evaluated the factor loadings shown in Table 4. For all items, factor loadings were stronger for the general environmental identity factor than the specific factors. This indicates that mean EID scale scores are appropriate. Next, we calculated Omega Hierarchical (ω_H) and ECV. The values obtained for these indices were .94 and .82, respectively. Additionally, McDonald’s Omega (ω) was .94, indicating that the EID scale items had an excellent internal consistency reliability.

Finally, to evaluate the consequences of adopting different models, we calculated correlations between environmental identity and connectedness with nature using several different approaches. CNS descriptive statistics are shown in the left-hand panel of Figure 3. First, we calculated Pearson’s correlations between the observed mean score across all EID scale items and the observed mean across all CNS items. As is shown in the right-hand panel of Figure 3, participants’ mean scores for the EID scale and CNS had a strong positive association: r = .66, 95% CI [.61, .70], p < .001. Next, we computed factor correlations based on the unidimensional model and the ESEM-bifactor model. All correlational results are summarized in Table 5. Regardless of the approach, all produced statistically significant positive correlations of similar magnitudes. As expected, the correlation based on observed means (r = .66) was smaller than the latent correlations; that is, measurement error lead to attenuated correlations. It was also noteworthy that the factor correlations for the ESEM-bifactor model (r = .75) and the unidimensional model (r = .77) were similar.

Past evidence about the latent structure of Clayton’s EID scale has offered contrasting findings. While some studies supported a multidimensional structure (Olivos & Aragonés, 2011), others identified, or at least assumed, a unidimensional structure (Chew, 2019; Clayton, 2003). Given this confusion in the literature, this study aimed to investigate competing factor models to offer clarity on the dimensionality of the EID scale.

First, we tested several competing models using CFA, including two plausible multidimensional models derived from the results of Olivos and Aragonés (2011), a unidimensional model, and a bifactor model. Our first major finding was that none of these models had a satisfactory fit to the data. While the bifactor model had the strongest fit in terms of CFI, RMSEA and SRMR, it was not interpretable due to a negative error term. The remaining three models did not demonstrate acceptable fit considering our predetermined thresholds (e.g., CFI values were all lower than .95), although the values for AIC, BIC, and SABIC suggested the first-order model (Model 1) had some predictive advantage. However, the very strong factor correlations in this model suggested issues with discriminant validity; that is, the model was likely forcing factors that were not present in the data. To investigate the structure of the EID scale further, we conducted ancillary analyses using a bifactor-ESEM approach. Unlike the predefined models, this bifactor-ESEM model presented excellent fit to the data. This finding suggests that all EID scale items represent a general environmental identity factor, but also that subsets of items share variance (representing nuisance effects) not accounted for by the general factor. Based on this finding, we suggest that past studies identifying multidimensional solutions (e.g., Olivos & Aragonés, 2011) may have done so, not because environmental identity is a multidimensional construct, but because the conceptual breadth of the construct results in diverse item content (Reise et al., 2000).

In addition to supporting the ESEM bifactor structure, we conducted several analyses that provide guidance on the best way to score and use the EID scale. Several findings helped us converge on the conclusion that it is appropriate for researchers to use the mean score across all EID scale items as an index of an individual’s environmental identity. First, factor loadings in the ESEM bifactor model were consistently higher on the general environmental identity factor than specific factors. Indeed, loadings on the specific factors were generally very poor (almost all lower than |.40|) meaning it was difficult to interpret the meaning of these shared sources of variance. Second, the values for omega hierarchical and ECV were high. Specifically, omega hierarchical indicated 94% of the variance in mean EID scale scores was attributable to the general environmental identity factor. Moreover, the ECV for the EID scale was .82, which implies factor loadings from a unidimensional model are a good approximation of factor loadings on the general factor in a bifactor model. Third, the fact that ω and ω_H were practically the same (difference of .003) indicated that the general factor of the ESEM bifactor model was the sole determinant of systematic variance associated with mean EID scale scores. Finally, we found that the relationship between environmental identity and connectedness with nature was largely unchanged by whether the EID scale was modeled as a unidimensional model or an ESEM bifactor model.

Combined, these findings offer evidence that EID scale items are essentially unidimensional and, therefore, capture a theoretically unidimensional construct. Put differently, our results showed that the nuisance effects of the specific factors in the ESEM bifactor model were mostly trivial. Consequently, we strongly advise against calculating individual subscale mean scores based on these specific factors. Indeed, these specific factors represent additional portions of variance not accounted for by the general factor and are not equivalent to theoretical dimensions of environmental identity. We also suggest strong caution interpreting any findings on the relation between EID subscales and outcomes because of the strong and inseparable contribution from the general factor. More positively, a key implication of this finding is that the EID scale appears to be sufficiently unidimensional that researchers can have confidence applying structural equation modeling (SEM) and IRT models that assume unidimensionality.

Despite the relevance of our findings, it is necessary to acknowledge limitations of the study. First, the study only used self-report data; a methodological choice that is criticized for introducing bias to data (e.g. social desirability effects). Second, the chosen sampling method has implications for the external validity of the findings. Non-probability sampling methods make it difficult to generalize research findings from a sample to the general population because they are characteristically non-random, meaning that is uncertain whether the present study findings would replicate in other Portuguese samples. A related issue is whether our findings, and the findings in past work, can be generalized outside of the ethnic and cultural contexts in which they are based. Cross-cultural studies testing measurement invariance are needed to determine if the EID scale measures the same essentially unidimensional construct in different cultural contexts. As far as we are aware, this has not yet been accomplished in the literature. Tests of measurement invariance are also important for determining whether the construct of environmental identity has the same meaning across other groups of individuals, such as men versus women. Other avenues for future research include applying the multitrait-multimethod matrix approach (Campbell & Fiske, 1959) using measures of other theoretically related variables, including those assessing pro-sustainable development behaviors, to provide a more robust assessment of the EID scale’s construct validity. Beyond the sphere of psychometric assessment, one necessary avenue for future research is to adopt longitudinal designs to describe (a) how environmental identity changes across important developmental periods (such as across adolescence), (b) the predictors of this change, and (c) the impact of such change on relevant outcomes, such as the adoption of pro-sustainable development behaviors.

To conclude, this study supports the use of the EID scale, and specifically the European Portuguese translation. We found an ESEM bifactor model had good fit to our data. In addition, based on (a) most items being more strongly related to the general factor than specific factors, (b) high values for omega hierarchical and ECV, and (c) finding that the relationship between environmental identity and connectedness with nature was largely the same for the unidimensional model and ESEM bifactor model, we conclude the EID scale captures an essentially unidimensional construct. Therefore, computing and interpreting a mean EID scale score is appropriate and preferable over computing means for individual subscales.

This work was supported by Portuguese Funds by Fundação para a Ciência e Tecnologia (FCT) [grant number UIDB/04375/2020].

The authors state that there are no conflicts of interest to declare.

Table S1. Factor loadings for the bifactor-CFA model. Docx

Raw participant data are available on the paper’s project page (available at: https://osf.io/ymb2v/).