Health Reports
Measuring workplace psychosocial factors in the federal government

by Ann-Renée Blais, Isabelle Michaud, Jean-François Simard, Lenka Mach and Simon Houle

DOI: https://www.doi.org/10.25318/82-003-x202101200001-eng

Abstract

Background

The National Standard of Canada for Psychological Health and Safety in the Workplace (the Standard) identifies 13 psychosocial factors affecting psychological health and safety in the workplace that employers should measure and monitor with the goal of addressing areas for improvement. The present study sought to determine the suitability of the Public Service Employee Survey as a tool for the assessment of these psychosocial factors in public service employees. It also aimed to explore—in a preliminary manner—predictors of job satisfaction in these employees.

Data and methods

Data from the 2017 and 2019 Public Service Employee Survey(PSES) were analyzed. Specifically, exploratory structural modelling and tests of measurement invariance were used to identify a measurement model reflecting the psychosocial factors outlined in the Standard and to evaluate the equivalence of this model across both PSES administrations.

Results

The analyses uncovered 10 of the 13 psychosocial factors, as well as 2 closely related factors—diversity and inclusion—and supported the full invariance of the resulting measurement model across both PSES administrations. Lastly, preliminary results pointed to recognition and reward (encompassing leader reward behaviours) and involvement and influence (touching on participative decision making, innovation and initiative) as predictors of job satisfaction in both samples of public service employees.

Interpretation

The present study identified the PSES as a tool for the assessment—in public service employees—of the majority of the psychosocial factors outlined in the Standard as well as two additional factors of particular importance to the federal government, diversity and inclusion. Future research to address current limitations is discussed, as are preliminary implications for practice.

Keywords

psychological health and safety, National Standard of Canada for Psychological Health and Safety in the Workplace, Public Service Employee Survey, public service employees, psychosocial factors, job satisfaction

Authors

Ann-Renée Blais (ann-renee.blais@statcan.gc.ca) is with the Human Resources Planning and Business Transformation Division, Statistics Canada, Ottawa. Isabelle Michaud and Lenka Mach are with the International Cooperation and Methodology Innovation Centre,  Statistics Canada, Ottawa.  Jean-François Simard is with the Statistical Integration Methods Division,  Statistics Canada, Ottawa. Simon Houle is with Concordia University, Montreal, Quebec.

Introduction

One in three Canadians will experience a mental illness in their lifetime.^{Note 1} Furthermore, approximately half a million Canadians are expected to miss work in a typical week because of such a problem or illness.^{Note 2} Because two-thirds of Canadians spend 60% or more of their time at work, the need for Canadian institutions and organizations, irrespective of their sector or size, to manage mental health issues within their workplaces is clear.^{Note 2Note 3}

Decades of research in several areas of the social sciences have further highlighted the importance of psychological health and safety (PHS) to organizations.^{Note 4Note 5Note 6} For example, employees who are psychologically healthy also tend to be more resilient, innovative, engaged and satisfied at work.^{Note 7Note 8Note 9Note 10Note 11Note 12} Conversely, reduced PHS has been linked to negative outcomes such as decreased productivity, absenteeism, turnover, increased health care expenditures and, in Canada alone, an annual economic burden estimated at $51 billion.^{Note 13Note 14Note 15Note 16Note 17Note 18} Moreover, providing a psychologically healthy and safe workplace is fast becoming a legal imperative, in light of recent developments (e.g., changes in labour law, employment standards and decisions related to human rights).^{Note 4}

The National Standard of Canada for Psychological Health and Safety in the Workplace

To provide systemic voluntary guidelines for Canadian employers regarding PHS, the Canadian Standards Association (CSA) Group and the Bureau de normalisation du Québec (BNQ) published the National Standard of Canada for Psychological Health and Safety in the Workplace (hereinafter referred to as the Standard), commissioned by the Mental Health Commission of Canada, in 2013.^{Note 4Note 19} With input from the social sciences, law and workplace health and safety scientific literature, the Standard aligns with decades of research in the fields of occupational health and organizational psychology.^{Note 20} The Standard identifies 13 workplace psychosocial factors affecting PHS that employers should assess and monitor with the objective of addressing areas for improvement.^{Note 4} These psychosocial factors are: Psychological Support, Organizational Culture, Clear Leadership and Expectations, Civility and Respect, Psychological Competencies and Requirements, Growth and Development, Recognition and Reward, Involvement and Influence, Workload Management, Engagement, Balance, Psychological Protection, and Protection of Physical Safety.^{Note 2Note 4}

The Job Demands-Resources Model (JD-R),^{Note 21Note 22} one of the leading job stress models,^{Note 23} assumes that any job demand (e.g., workload, downsizing) and any job resource (e.g., balance, psychological support) may affect employee health and well-being through the integration of two basic psychological processes.^{Note 23} First, excessive job demands and deficient job resources trigger a “stress process,” which may, via burnout (an intervening variable or mediator of this process), bring about negative outcomes for both the employee and organization (e.g., health symptoms, poor job performance). Second, abundant job resources spark a “motivational process” that may, through engagement (a mediator of this process), generate positive outcomes in turn (e.g., job satisfaction, superior job performance). The JD-R Model appears particularly well suited to the examination of the relations between the aforementioned psychosocial factors, all of which would be considered job resources in this framework,^{Note 23} and their influence on PHS.^{Note 20} Ivey and colleagues engaged in this exercise by applying a JD-R lens to a theoretical model of PHS including the 13 psychosocial factors, burnout and engagement, and negative individual and organizational outcomes (i.e., psychological distress and turnover intentions) with the objective of providing a conceptual framework to guide empirical research and workplace interventions in turn.^{Note 20}

In 2016, the Government of Canada adopted the Federal Public Service Workplace Mental Health Strategy (hereinafter referred to as the Strategy),^{Note 3} which reflects the government’s commitment to establishing a healthy, respectful and supportive federal workplace—a commitment the Clerk of the Privy Council re-emphasized in his twenty-fifth Annual Report.^{Note 24} Recognizing that mental health constitutes a key dimension of an individual’s wellness, one of the strategic goals of the Strategy is to measure and report on mental health in the workplace.^{Note 3} Incidentally, the Strategy refers to the Standard as an effective framework for guiding employers’ efforts to promote PHS in the workplace.^{Note 3}

Thus, with the Standard as a starting point, the main objective of the present study was to develop a measurement tool that enables the improvement of PHS for public service employees. To date, no research has been published in the scholarly literature that systematically documents the development of a tool reflecting the 13 psychosocial factors, with one exception: Ivey and colleagues, using multiple-item scales drawn from the scientific literature, assembled a comprehensive survey that aligns with the Standard—the Unit Morale Profile version 2.0 (UMP v2.0)—for the Department of National Defence and the Canadian Armed Forces.^{Note 20}

Despite the UMP v2.0’s strengths,^{Note 20Note 25} its administration to employees across the public service would be a complex endeavour. Moreover, the 141-item UMP v2.0 is a lengthy survey, which may also limit its widespread utilization in this context. As a result, and given the availability and wide reach of the Public Service Employee Survey(PSES),^{Note 26} Arim and colleagues relied on it to identify a subset of items capturing the 13 psychosocial factors outlined in the Standard, while recognizing that this was not the intended purpose of the PSES.^{Note 27} This limitation notwithstanding, Arim and colleagues were able to distinguish—through both a concept-mapping exercise and factor analyses—9 of the 13 factors in a sample of public service employees using the 2017 PSES.^{Note 27}

The present study

Expanding on the work of Arim and colleagues,^{Note 27} the present study sought first to explore a different set of PSES items, that is, the set of items common to both the 2017 and 2019 PSES,^{Note 28Note 29} as potential indicators of the 13 psychosocial factors identified in the Standard. This specific set of PSES items is hereafter referred to as the Public Service Psychosocial Questionnaire (PSPQ) in order to distinguish it from the PSES as a whole. Focusing on the items common to both the 2017 and 2019 PSES allows for the investigation of the psychometric properties of the retained items across two cross-sectional samples of public service employees.

Second, echoing the JD-R Model, the present study aimed to examine preliminary links between the uncovered psychosocial factors and job satisfaction. Job satisfaction, defined as a pleasurable or positive emotional state resulting from the appraisal of one’s job or job experiences (p. 1300),^{Note 30} is one of the most important and widely researched outcomes in organizational psychology.^{Note 30} Meta-analyses have shown moderate-to-strong correlations between an abundance of job resources (e.g., autonomy, clarity and positive leadership) and high job satisfaction,^{Note 31Note 32} as well as small-to-moderate correlations between job satisfaction and several of its correlates and presumed consequences, such as health symptoms and job performance.^{Note 32}

Specifically, the present study posed the following research questions:

Research Question 1 (RQ1):Is the PSPQ a suitable tool for the assessment of the 13 psychosocial factors outlined in the Standard?

Research Question 2 (RQ2): Are the psychometric properties of the PSPQ equivalent across two samples of public service employees?

Research Question 3 (RQ3): Are the psychosocial factors predictors of job satisfaction?

Data and methods

Participants

The PSES, a census with voluntary participation, assesses public service employees’ perceptions of their leadership, workforce, workplace, workplace well-being and compensation in the first official language (FOL) of their choice.^{Note 28} The goal of this survey is to inform the continuous improvement of people management practices in the federal government.^{Note 33} From 1999 to 2017, the PSES was conducted triennially and since 2018, it has been administered annually to active employees in organizations in the federal public service and in participating separate agencies.^{Note 33} The PSES has a cross-sectional design, and data collection mainly takes place electronically.^{Note 28} Respondents rate the vast majority of the PSES items on a five-point scale ranging from 1 (Strongly Agreeor Always) to 5 (Strongly Disagreeor Never). In the present study, the ratings were reverse-coded where needed.

In 2017, the 119-item PSES, led by the Office of the Chief Human Resources Officer (OCHRO), in collaboration with Statistics Canada, was available for completion from August 21 to September 29.^{Note 28} A total of 174,544 employees in 86 federal departments and agencies responded to the survey for a response rate of 61%.^{Note 28} Of the respondents, 57% were female (41% male, 2% other or missing), 64% were 40 or older (35% younger than 40, 1% missing) and 68% reported English as their FOL (31% French, 1% missing).

In 2019, the 112-item PSES, administered by Advanis on behalf of OCHRO, was available for completion from July 22 to September 6.^{Note 26} A total of 182,306 employees in 86 federal departments and agencies responded to the survey, for a response rate of 62.3%.^{Note 26} Of the respondents, 58% were female (40% male, 2% other or missing), 61% were 40 or older (38% younger than 40, 1% missing) and 69% reported English as their FOL (31% French, less than 1% missing).

Measures

Eighty-nine items, common to the 2017 and 2019 PSES, were identified to ensure continuity in measurement. Items irrelevant to the purpose of the present study and sociodemographic questions were not included in this initial set of items. A panel of five experts (four research psychologists and a practitioner familiar with PHS in the workplace) examined these items to inform content validity for theoretical fit to the 13 psychosocial factors, as well as to an additional factor, diversity and inclusion (D&I). This last factor, albeit not a recognized psychosocial factor in the Standard, was also investigated in the present study in recognition of its importance in a healthy workplace.^{Note 34}

Based on the factor definitions, the experts were asked to map each of the retained items onto one of the psychosocial factors. They also had the option of assigning an item to an “Other” category if they felt this item did not map onto any of the factors. A majority of experts agreed on 78% of the items, resulting in a moderate inter-rater agreement of 0.450 (SE = 0.010) across items, as measured by Fleiss’s kappa. A z test rejected the hypothesis of chance agreement among experts (p < 0.001). Each item was assigned to a factor based on the average pairwise agreement among experts.^{Note 35}

None of the experts mapped any items onto the psychological competencies and requirements factor and they assigned only one to the engagement factor. Consequently, these two psychosocial factors were not further considered in the current study. As a result of the conceptual mapping exercise, 23 items were dropped from further analyses, reducing the pool of items from 89 to 66 with anywhere from 2 to 15 mapped items per factor.

Statistical analyses

A 12-factor exploratory factor analysis (EFA) with oblique target rotation was used to examine the 66 items’ standardized factor loadings on their inferred factor. Target rotation offers researchers the possibility of testing more theoretically appropriate solutions by stipulating a priori which items ought to load on what factors.^{Note 36} Here, the experts’ mapping of the PSES items onto the psychosocial factors guided the specification of the factors and target loadings. Through a process of pruning items that poorly reflected their hypothesized factor (i.e., items with a standardized target loading lower than 0.30 and one or a few cross-loadings),^{Note 37} a viable 43-item solution was retained (for these items specifically, Fleiss’ kappa reached 0.571 [SE = 0.015], p < 0.001).

This final EFA solution revealed 12 properly identified oblique factors with at least 2 acceptably high standardized target loadings (i.e., greater than or equal to 0.30). Contrary to expectations, the postulated psychological support factor failed to emerge and the D&I factor split into separate yet correlated diversity and inclusion factors. Following this examination of the data, the final EFA model was converted into an exploratory structural equation model (ESEM),^{Note 38} to evaluate the invariance of the measurement model across the 2017 and 2019 PSPQ.

All EFA and ESEM models were estimated using the robust weighted least square estimator with mean and variance adjusted statistics (WLSMV) implemented in Mplus 7.4,^{Note 39} applying the survey weights for the PSES (28) and accounting for the clustering of respondents within departments and agencies.^{Note 39} This estimator is best suited to the ordinal nature of the Likert-type scales and asymmetric response thresholds of the instruments used in the present study.^{Note 40Note 41Note 42Note 43} Partial item non-response was handled prior to the analyses using a donor imputation technique.^{Note 44}

The WLSMV chi-square test of exact fit (χ²), the comparative fit index (CFI), the Tucker-Lewis index (TLI), the root mean square error of approximation (RMSEA) and its confidence interval were examined.^{Note 45Note 46} However, only the sample-size independent goodness-of-fit indices (i.e., CFI, TLI, RMSEA) were used to empirically assess model fit because of the known oversensitivity of the χ² to sample size and minor sources of misfit.^{Note 45Note 46} CFI and TLI values greater than 0.90 and 0.95, respectively, and RMSEA values lower than 0.08 and 0.06 indicate an adequate and an excellent model fit, respectively.^{Note 45Note 47} In tests of measurement invariance,^{Note 48} a decrease in CFI and TLI of more than 0.01 or an increase in RMSEA of more than 0.015 signifies a lack of support for the invariance of the measurement models.^{Note 49Note 50} The tests of measurement invariance, adapted to WLSMV estimation, were conducted in the following sequence:^{Note 48Note 51} (1) configural invariance (i.e., same model across samples), (2) weak invariance (i.e., identical factor loadings), (3) strong invariance (i.e., identical response thresholds), (4) strict invariance (i.e., identical item uniquenesses), (5) invariance of the correlated uniquenesses, (6) invariance of the latent variance-covariance matrix, and (7) invariance of the latent means.

Results

Sample 1: The 2017 PSPQ

The goodness-of-fit indices for the ESEM model, reported in Table 1, indicate an adequate fit to the 2017 PSPQ data.

Sample 2: The 2019 PSPQ

The preceding findings suggest that the PSPQ, with its subset of 43 items originating from the PSES, captures 10 of the 13 psychosocial factors outlined in the Standard, as well as 2 additional factors, diversity and inclusion. However, the only way to determine whether or not its psychometric properties generalize to the population of public service employees is to systematically evaluate the extent of this generalization on a new sample drawn from this population. Accordingly, the ESEM model was applied to the 43-item PSPQ in Sample 2.

The goodness-of-fit indices for the ESEM model, reported in Table 1, indicate an adequate fit to the 2019 PSPQ data. To systematically evaluate the extent to which the parameter estimates obtained in Sample 1 generalized to Sample 2, the invariance of the ESEM solution was examined across samples. The results from this examination, also shown in Table 1, demonstrate its complete invariance across samples (i.e., the invariance of its factor loadings, intercepts, uniquenesses, correlated uniquenesses, latent variances and covariances, and latent means).

Relations with job satisfaction

The associations between the emergent psychosocial factors and job satisfaction were tested across samples. An item reflecting job satisfaction was added to the model of latent means invariance and regressed on the factors across samples through an ordered probit regression model, the default with the WLSMV estimator in Mplus.^{Note 39} In a subsequent model, these regressions were constrained to equality.^{Note 52} The goodness-of-fit of these two models, included at the bottom of Table 1, substantiates the equivalence of the relations between the factors and job satisfaction across samples. The job satisfaction item, “Overall, I like my job” (see Wanous and colleagues for a review of the appropriateness of single items to assess overall job satisfaction)^{Note 53}, was rated on a 5-point scale ranging from 1 (Strongly Agreeor Always) to 5 (Strongly Disagree or Never).

The parameter estimates from the ordered probit regression model are shown in Tables 2 and 3. For parsimony, only shortened item stems are included in Table 2, but the complete wording of the PSES items is publicly available.^{Note 29} On average, the factors present a satisfactory level of composite reliability (0.79, ranging between 0.45 and 0.96), with the exception of the two-item diversity factor (0.45). They are defined by high standardized target loadings on average, 0.68, ranging between 0.29 and 0.99, p < 0.001. The factor correlations, given in Table 3, are moderate on average,0.38, ranging between 0.08 and 0.73, p < 0.001. The probit regression coefficients are reported in Table 4, where the unstandardized coefficients represent the change in the job satisfaction probit index for a one-unit change in the predictor (e.g., a one-unit change in involvement and influence was related to a 0.365 increase in the job satisfaction probit index). Although all of the psychosocial factors were significantly related to job satisfaction (pseudo R² = 0.604, SE = 0.003, p < 0.001), the recognition and reward and involvement and influence factors showed moderate associations with job satisfaction.

Discussion

Summary

Based on the results of the present study, the 43-item PSPQ, originating from the PSES, constitutes a way to assess 10 of the 13 psychosocial factors outlined in the Standard as well as 2 closely related factors, D&I, in public service employees (RQ1). The findings also support the full measurement invariance of this measurement tool across two cross-sectional samples of public service employees, providing preliminary evidence for its generalizability (RQ2). Lastly, preliminary analyses point to the recognition and reward and involvement and influence factors as moderate predictors of job satisfaction in both samples (RQ3). These findings, albeit preliminary, align with past research: the recognition and reward factor encompasses leader reward behaviours, and the involvement and influence factor alludes to participative decision making, innovation and initiative.^{Note 31Note 32} Despite this initial evidence, however, future work should replicate these results with a more fulsome measure of job satisfaction prior to making recommendations for workplace interventions in the public service.

Limitations and directions for future research

First and foremost, the current study employed a rather counter-intuitive methodological sequence in terms of mapping pre-existing survey items onto the 13 psychosocial factors and D&I, rather than benefiting from the traditional approach of selecting items from the broader academic and applied literature to represent the pre-established factors. In addition, the moderate inter-rater agreement raises concerns about the content validity of the resulting measurement tool and the iterative process through which the final items were selected. Although necessary for the needs of the study, there is a possibility that the measurement model may be over-fitted to the data.^{Note 54} This prospect is mitigated by the fact that cross-sectional measurement invariance was obtained, thus replicating the model in a new sample. Nevertheless, further replication of the factor structure of the PSPQ over time should be conducted before putting this tool to widespread use.

Self-reported data were used exclusively, potentially leading to social desirability and biases. Future research should include objective criteria (e.g., absenteeism, turnover) to mitigate this shortcoming. In particular, the potential for integrating self-reported data with data in human resources databases should be considered. For example, linking the psychosocial factors to outcomes such as absenteeism would allow for the identification of predictors of these outcomes and for the estimation of productivity costs. This type of research endeavour would also provide sizeable construct validity evidence by avoiding common method bias.^{Note 55}

The present study was based on cross-sectional designs, thereby making it impossible to reach clear conclusions regarding the probable causal effects of the psychosocial factors on job satisfaction. Future studies would gain from a longitudinal design and the examination of the directionality of the relationships between the psychosocial factors and their purported predictors and outcomes. Considerable work also remains to identify appropriate items reflecting the three psychosocial factors that are not currently included in the PSPQ: Psychological Support, Engagement, and Psychological Competencies and Requirements, as well as an additional diversity item to create a version of the PSPQ with at least three items capturing each factor. Importantly, to provide additional support for the construct validity of the PSPQ scores, correlations with scores on established measures to which they are theoretically related (e.g., contingent rewards facet of the Job Satisfaction Survey, Global Transformational Leadership scale), as well as with a variety of criteria (e.g., burnout, psychological distress, absenteeism, turnover), should be tested.^{Note 56Note 57}

Another limitation has to do with the Standard itself. Kunyk and colleagues, through focus groups of employers, found that although the Standard resonates with organizational values and beliefs and can provide direction for positive change, the scope and complexity of the 13 psychosocial factors and the document outlining the Standard can present challenges to organizations.^{Note 58} Supporting these qualitative findings, the present study highlighted not only content validity concerns with the mapping of the PSES items onto the psychosocial factors, but also moderate factor correlations, suggesting—if not a lack of discriminant validity (i.e., factor correlations greater than |0.85|)^{Note 59}—at least some degree of redundancy among the factors. Future work might seek to extract and focus on a core set of factors that emerge as consistent empirical and theoretical predictors of several pertinent outcomes as a way of limiting the considerable effort and complexity associated with measuring, reporting on and continuously improving on all factors.

Implications for practice

The current study illustrates how workplace psychosocial factors can bolster positive individual and organizational outcomes such as job satisfaction. Knowing what “levers” to apply in order to improve these outcomes can inform the design of interventions. These interventions should utilize psychometrically sound measures, adequate experimental designs and sufficiently large samples.^{Note 60} Michie and Williams emphasized the need for interventions based on human resources management practices.^{Note 60} Such interventions would exemplify primary prevention (e.g., reinforcing workplace resilience factors associated with psychological health) rather than secondary prevention (e.g., training of managers to identify employees in distress and orient them to appropriate treatment).^{Note 60}

Conclusion

The present study identified the PSPQ, comprised of 43 PSES items, as a tool for the assessment of the psychosocial factors defined by the Standard in public service employees. Furthermore, it drew attention to the role these factors can play in predicting individual and organizational outcomes and guiding workplace interventions in turn.