- Research article
- Open Access
Evaluate construct validity of the Revised American Pain Society Patient Outcome Questionnaire in gynecological postoperative patients using confirmatory factor analysis
BMC Anesthesiology volume 21, Article number: 20 (2021)
The Revised American Pain Society Patient Outcome Questionnaire (APS-POQ-R) evaluates the patient-reported quality of pain management in adults. A validated APS-POQ-R is pivotal to guide effective pain management with better patient satisfaction. Previous studies revealed that subscales of “patients’ perception of pain management” were unstable cross-culturally. This study aims to evaluate the construct validity of the APS-POQ-R in gynecological postoperative patients with a multi-cultural background using confirmatory factor analysis to allow comparisons among different a priori models at the latent factor level.
Patients aged 18 years old or above and who were scheduled for gynecology surgery were selected. Three different models with a combination of latent factors were based on a priori hypotheses from previous studies. The root-mean-squared error of approximation, comparative fit index, Tucker-Lewis Index, Chi-squared test, and change in Chi-squared statistic given a change in degrees of freedom between models were used to assess the model fit to the present data.
A total of 302 patients completed the questionnaire. The five-factor model which was based on Gordon’s study has an acceptable fit for the data and was superior when compared to the one-factor baseline model. Although the four-factor model, which originated from Botti’s study, also demonstrates a good model fit, the “perception of care” construct was excluded in this model. The “perception of care” construct is conceptually important as patient-centered care has become the focus of quality improvement of pain service.
The APS-POQ-R is easy to administer and is useful for quality evaluation in postoperative pain management. The present study demonstrates that a five-factor structure of the APS-POQ-R is the best fitting model in our patient sample. The results of this study provide further evidence to support the use of APS-POQ-R as a measurement tool for pain management evaluation in acute postoperative patients with a multi-cultural background.
Pain is an inevitable consequence of surgery, and managing pain is a challenging task as pain is a subjective experience and multi-dimensional . Postoperative pain is inadequately managed in more than 80% of surgical patients, and is associated with increased morbidity, impaired recovery from surgery, and decreased quality of life . Besides monitoring of clinical outcomes, continual evaluation of the patient-reported outcome in the quality of pre- and post-operative pain management is pivotal to guide an effective health care delivery with a high level of patient satisfaction. For this purpose, a validated and standard quality improvement (QI) measure is essential.
The Acute Pain Society Patient Outcome Questionnaire (APS-POQ) was first developed in 1991 , and was subsequently revised in the years 1995  and 2010 by the American Pain Society . The Revised Acute Pain Society Patient Outcome Questionnaire (APS-POQ-R) could be a useful tool to measure the quality of pain management for QI purposes. The APS-POQ-R was originally organized into five factors in the psychometric evaluation by Gordon et al.: pain severity and sleep interference (5 items), activity interference (2 items), affection (4 items), adverse effects (4 items), and perceptions of care (3 items). While the original factor solution in that study showed a high degree of internal consistency, subsequent studies in various cohorts reviewed a range of factor solutions and patterns of item distribution [6,7,8].
Botti et al. divided the primary questionnaire items that were amenable to psychometric testing into two main categories based on their face meaning; namely, pain experience and patients’ perception of pain management. The “pain experience” category measures the pain severity, interference of physical and psychological well-being due to pain, and the “patients’ perception of pain management” category evaluates patients’ participation in decision making and their satisfaction . The authors confirmed the validity and stability of APS-POQ-R to measure “pain experience” across different cultures. Nevertheless, the robustness of APS-POQ-R to measure “patients’ perception of pain management” across cultural groups was questioned due to the inconsistent loading of “pain management” items between different groups in their study. It is believed that the reasons for the variability in the perception of treatment across national groups are attributed to the difference in culture, health care delivery, language, and translation .
This finding was echoed in a study by Zoega et al. who evaluated the Icelandic version of the APS-POQ-R . Although the initial principal components extracted in that study were consistent with those from Gordon et al. , the internal consistency obtained was unacceptable with a Cronbach α value of 0.42. To increase the internal consistency, Zoega et al. removed the perception of the care component in the final Icelandic version of APS-POQ-R . The cross-cultural instability of APS-POQ-R to measure “patients’ perception of pain management” suggests the importance of conducting a validation evaluation of this questionnaire before implementing its use in a different regional setting.
From the literature, the models that were proposed for APS-OQ-R may have three-, four-, or five-factor structures. The differing patterns of item distribution and removal of items in previous studies render the questionnaire difficult to compare findings between cohorts [8, 9]. In this study, we aim to evaluate the construct validity of this instrument using the confirmatory factor analysis (CFA) framework of the previously proposed models in a multi-ethnic Malaysian population, focusing on patients receiving gynecological procedures. Confirmatory factor analysis, unlike exploratory factor analysis (EFA), allows comparisons of alternative a priori models proposed at the latent factor level to determine the best fitting model that describes the data . Gynecological postoperative patients represent a large proportion of surgical patients and there is increasing evidence on sex differences in pain experience, and analgesia with women report greater pain in acute clinical pain settings . With a validated QI instrument, we can explore the multi-dimension of pain experience in this group of patients and develop targeted pain management .
This cross-sectional study was carried out at the University of Malaya Medical Centre (UMMC), a tertiary university hospital in Kuala Lumpur, Malaysia. The present study complies with the Declaration of Helsinki and the study protocol was approved by the UMMC Medical Research and Ethics Committee (Approval number: 2,016,1224660), as well as the International Medical University Joint Committee on Research and Ethics (Approval number: IG544), Kuala Lumpur, Malaysia. Written informed consent from patients was obtained before their participation.
In our center, postoperative pain management is managed by both the Acute Pain Service (APS) team, and the primary surgical team. The usual postoperative multimodal analgesia regimens are a combination of opioids and non-opioid analgesics. The commonly used opioids include fentanyl, morphine, and oxycodone; while non-opioid analgesics are paracetamol, non-steroidal anti-inflammatory drugs (NSAIDs), and local anesthetic agents. Regional anesthetic techniques will be considered if indicated.
Patients aged 18 years old or above and who were scheduled for elective gynecological surgery were eligible to be enrolled in this study. All elective surgeries were scheduled via the hospital’s online operation booking system in our center. A total of 320 patients were identified from the booking system. The investigator who was not directly involved in the management of the patients would communicate with them and introduce the APS-POQ-R questionnaire after they agreed to participate. A patient’s participation was voluntary, and standard care was ensured for all patients. We excluded the patients who had unexpected postoperative complications or intensive care admission as they were unable to answer the questionnaire.
The APS-POQ-R measures the quality of postoperative pain management among adults in the first 24 h of in-hospital patient care, and the questionnaire can be administered by interview or self-completion . This questionnaire has been translated into 11 different languages and is widely used for QI purposes in pain management . The APS-POQ-R can be obtained from the American Pain Society freely and can be used without further permission.
The questionnaire contains a total of 23 items which include 18 primary items (P1 – P9) and 3 secondary or information items (P10 – P12). The first three primary items (P1 – P3) measure patients’ pain intensity. Patients are asked to rate the least and the worst pain that they have in the first 24 h postoperatively on a numerical rating scale (NRS) ranging from 0 (No pain at all) to 10 (the worst pain possible). Meanwhile, item P3 examines the amount of time that patients experienced pain on a 0 to 100% scale during their first postoperative day.
Item P4 of the APS-POQ-R examines the interference of postoperative pain on patients’ routine functioning which includes activities in and out of bed, as well as the impact of pain on their sleep. Also, item P5 assesses the effects of pain that may result in anxiety, depression, fright, or helplessness on postoperative patients. Treatment-related adverse effects are an important component of pain management. Item P6 elicits the severity of side effects associated with analgesics. Insights regarding the side effects associated with postoperative pain management will help identify areas of improvement in the current practice.
Patients’ participation in the management of their care is of paramount importance to ensure comfort and to reduce potential complications . The patients’ perception of the extent to which they can participate in decision-making in pain management is assessed in item P7. Item P8 grades the patients’ satisfaction in pain treatment received during the postoperative period. All items from P1 to P8 are measured with a numeric rating scale from 0 to 10, except for Item P3 and Item P7 which are measured in percentage.
The additional assessments on non-pharmacologic pain management include the items that evaluate whether the patients receive any information about other options for pain treatment, and how useful the information is to them (Item P9). The use of non-pharmacologic methods to relieve pain, and whether a nurse or a doctor encourages the patients to use non-pharmacologic methods are assessed in Item P10 and Item 11, respectively. These items give secondary information and are not included in the psychometric testing. The information on the patient characteristic profile is also obtained from the questionnaire.
The study was carried out between June 2017 and June 2018 using APS-POQ-R (English version). An investigator would approach potential study subjects preoperatively to obtain their informed consent. On a postoperative day, the consented patients were then asked to recall the pain management that they received in the first 24 h postoperatively. The demographic and clinical data of these patients were retrieved from their respective electronic medical records.
Confirmatory factor analysis
We used confirmatory factor analysis (CFA) to evaluate the construct validity of the APS-POQ-R. The CFA was computed using maximum likelihood estimation in AMOS version 26™ (SPSS, IBM, Inc.). Confirmatory factor analysis is a subset of structural equation modeling (SEM), whereby it focuses on analyzing the extent to which the observed variables are related to the latent factors (measurement model); while SEM analyzes the causal relationship between observed variables and latent factors (structural model) . Previous studies on APS-POQ-R utilized an exploratory factor analysis approach and showed various factor solutions in their respective settings. Confirmatory factor analysis reduces the numbers of observed variables to latent factors based on communalities and allows for comparisons of different a priori models at the latent factors level . Using the CFA framework is suitable to fulfill the purpose of the present study: to find and confirm which constructs fit the data from our patient cohorts.
Multi-trait multimethod matrix (MTMM) is another statistical tool to evaluate construct validity. This analysis is used when the data set involves multiple constructs that are measured by different methods, such as self-report questionnaires, investigators’ observation, and so on . Our study, however, used patient-report questionnaires to measure the underlying constructs. Thus, MTMM is not utilized in this study.
The models tested were based on the a priori hypothesis from previous studies [5, 6]. Four competing models were tested in the analysis. Model I, a single-factor model in which all 18 items were loaded on the latent variable, was used as a baseline comparison against the other models.
Model II, the original classification by Gordon et al. , was a five-factor model with items P1-P3 and P4c-P4d loaded on “Pain severity and sleep interference subscale”; items P5a-P5d loaded on “Affective subscale”; items P4a-P4b loaded on “Activity interference subscale”; items P6a-P6d loaded on “Adverse effect subscale”; and lastly, items P7-P9 loaded on “Perception of care subscale”. Model III was composed of four factors with items P1-P3, P4a-P4b, P7, and P9 loaded on “Pain severity/pain care and activity interference subscale”, P4c-P4d loaded on “Sleep interference subscale”, P5a-P5d loaded on “Affective subscale”, and P6a-P6d loaded on “Adverse effects subscale”. This model excludes item P8 (Ability to participate in pain treatment) that was based on the finding from Botti et al. , in which P8 did not load on any factors in the Danish cohort. Model IV, which was also a model based on Botti’s study , is a three-factor model with items P1-P3, P4a-P4d, and P7 loaded on “Pain severity/pain care and activity interference extended subscale”; P5 (a-d) and P6 (a-d) loaded on “Affect and adverse effects subscale”; and items in P8-P9 loaded on “Perception of care subscale”.
The assessment of model fit was based on several fit indices to test the best CFA model that describes the present data set and theoretical considerations. These indices include the root-mean-squared error of approximation (RMSEA), comparative fit index (CFI), Tucker-Lewis Index (TLI), Chi-squared test, and a change in Chi-squared statistic given a change in degrees of freedom between models. The RMSEA values closer to 0 with a threshold of less than 0.08 represent a good model fit . The CFI and TLI values of 0.95 and above were considered indicative of an excellent fit; and above 0.9, is acceptable .
There is no easy number that one can use as a “large enough” sample size for structural equation modeling. Some rules of thumb are used; based on Kline et al., 100–200 observations are considered a medium sample size . Different ratio rules are recommended, but for most multivariate analyses, the sample size should be at least 10 times the number of variables [18, 19]. With 18 items in the questionnaire that are subjected to factor analysis, a sample size of 320 is considered sufficient for psychometric testing in the present study.
Sample descriptive statistics
Descriptive statistics were used to describe the demographic data, and to report the results of the questionnaire items. Descriptive statistics were performed by using the Statistical Package for the Social Sciences, version 21.0 (SPSS Inc, Chicago, USA), and MS Excel (2013).
Of the 320 eligible patients in this study, 11 patients refused to participate, and seven returned questionnaires were incomplete. This gave a response rate of 94.4% (302/320). A flowchart of the patient recruitment process based on STROBE guidelines  is illustrated in Fig. 1. The demographics of the participants are depicted in Table 1.
Confirmatory factor analysis
The fit statistics of all models are shown in Table 2. Figure 2 depicts the one-factor model as the base model. The model fit of the five-factor structure (Model II) as described in Gordon’s study fulfills an acceptable fit criterion (Fig. 3). The Chi-square score relative to degrees of freedom (χ2/df) is small (2.47), while the Comparative Fit Index (CFI) and Tucker-Lewis Index (TLI) are within an acceptable range. The root-mean-squared error of approximation (RMSEA) of this model also indicates an adequate fit.
Model III (Fig. 4) was based on the four-factor solution from the study by Botti et al.  also demonstrates an acceptable model fit with small χ2/df (2.40), CFI of 0.922, TFI of 0.895, and RMSEA of 0.068. The fit indices for the three-factor model (Model IV), however, were not acceptable based on this dataset (Fig. 5).
Factor loadings for each of the latent factors for Model II are shown in Table 3. Three items had a factor loading of less than 0.40. These items are “Estimate of time in severe pain” and “Itchiness” for both models, as well as “Participation in decision making of pain treatment” in Model II.
Descriptive data of APS-POQ-R
The items that measure the responses with a numerical rating scale from 0 to 10 were considered as continuous variables. The number of respondents, minimum, maximum, mean, and standard deviations (SD) of the pain scores are shown in Table 4.
The present study examines the construct validity of the APS-POQ-R using confirmatory factor analysis, in contrast to the use of exploratory factor analysis in previous studies. Our findings support the five-factors structure that was hypothesized by Gordon et al.  in a multi-ethnic patient sample using the CFA approach.
The psychometric evaluation of APS-POQ-R has been performed in adult inpatients across different cultures using an exploratory factor analysis approach [7, 8, 21]. The construct validity for subscales “pain severity”, “adverse effects”, and “interference of activity and sleep” was consistent across different cohorts. The items “percentage of pain relief received”, “participation in decision making” and “satisfaction”, however, have inconsistent loadings on the latent factors across studies . These three items were proposed to measure the “quality of pain management” domain from the original study by Gordon et al. . The poor psychometric properties of the “quality of pain management” domain suggest that the validity and reliability of APS-POQ-R need to be evaluated before being implemented in a local clinical setting .
Among the hypothetical models tested in this study, two models (model II and model III) demonstrate acceptable fit indices and are superior to the one-factor model. Although Model III was based on the study by Botti et al.  has an overall satisfactory model fit, the latent factor of “Perception of care” is lacking. Moreover, the item “ability to participate in pain treatment” was not included in model III due to its low communality in the original study. The authors, however, suggested to retain the item and considered it as independent of APS-POR-R constructs due to its conceptual importance .
In recent years, patient-centered care has been a focus in the healthcare system, and patients’ needs are considered in the decision-making process . Patients’ perception of care, therefore, becomes an integral component in the development of questionnaires for quality improvement purposes. As such, Model II that includes all items that are proposed to measure patients’ satisfaction and perception would be a better model than other models for the APS-POQ-R construct.
For Model II, the item “participation in decision making of pain treatment” has a standardized factor loading of less than 0.3 in our cohort. The interpretation of “participation in pain management” may be different among individuals. McTier et al. reported that most patients tended to report their pain severity rather than participating in decision-making on the treatment options . Some patients may perceive that reporting pain is a form of participation, but others may prefer choosing a treatment option for themselves. To improve the representativeness of this item for the underlying latent factor, modification of the item may be considered.
The “affective” and “adverse effects” subscales are stable across previous studies . The standardized factor loading for “itchiness”, however, was lower than that of other adverse effects. A few reasons may explain this finding. First, there was a high proportion of patients who did not experience this adverse effect . Second, different analgesics administered to the patients may affect the manifestation of the side effects. For example, dizziness, nausea, and vomiting, as well as itchiness, are more common in opioids users than those who received paracetamol or non-steroidal anti-inflammatory drugs. The difference in the combination of analgesics administered to the patients may contribute to the lower factor loading for “itchiness”.
The satisfaction score among our participants was high, with a mean value of 8.05. The influence on the level of satisfaction of postoperative pain management could be multifactorial. The level of patient satisfaction does not only depend on the effectiveness of pain relief but is also determined by the care from the healthcare providers. An opportunity to participate in pain treatment and to gain better knowledge about the pain care that they are receiving may contribute to higher patient satisfaction [21, 25].
Our findings support the APS-POQ-R as a QI tool for pain management evaluation in this setting. This validated questionnaire can be used as a standard tool to provide a benchmark for the comparison of pain management quality within the measured patient cohort. Besides, the different subscales of the questionnaire allow clinicians or researchers to identify areas for improvement more effectively. Of note, we plan to utilize this questionnaire for our postoperative patients to enable continual evaluation and reassessment of the construct stability over time. The use of patient-reported pain experience measures will also encourage patients’ engagement in their pain care. In the future, the psychometric testing of APS-POQ-R using the CFA approach should be performed in other clinical settings such as medical patients who receive pain care to establish a standardized QI instrument for comparison of the quality of pain care.
Our study has a few limitations. First, the patients may suffer from the lingering effects, such as dizziness and sleepiness of anesthesia in the postoperative period. Hence, the recall of postoperative pain experience may be affected. Second, we recruited only gynecological patients who represent one of the main surgical subspecialties in our center. Recruitment of patients from different subspecialties in the future would enable the generalization of the study results to other surgical patients.
The psychometric testing of a measurement instrument is important to support its use in research or clinical practice. The present study has demonstrated that a five-factor structure of the APS-POQ-R is the best fitting model in our multi-ethnic patient sample. This study results provide further evidence to support the use of APS-POQ-R as a measurement tool for pain management evaluation.
Availability of data and materials
The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request.
Revised American Pain Society Patient Outcome Questionnaire
Confirmatory factor analysis
Exploratory factor analysis
University of Malaya Medical Centre
International Medical University
Acute pain service
Non-steroidal anti-inflammatory drugs
Root-mean-squared error of approximation
Comparative fit index
Dihle A, Helseth S, Christophersen KA. The Norwegian version of the American Pain Society Patient Outcome Questionnaire: reliability and validity of three subscales. J Clin Nurs. 2008;17(15):2070–8.
Gan TJ. Poorly controlled postoperative pain: prevalence, consequences, and prevention. J Pain Res. 2017;10:2287–98.
Bond MR, Charlton JE, Woolf CJ, editors. American Pain Society Committee on quality assurance standards: American Pain Society quality assurance standards for the relief of acute pain and cancer pain. Proceedings of the VIth world congress on pain; 1991; Seattle: IASP Press.
American Pain Society Quality of Care Committee. Quality improvement guidelines for the treatment of acute pain and cancer pain. JAMA. 1995;274(23):1874–80.
Gordon DB, Polomano RC, Pellino TA, Turk DC, McCracken LM, Sherwood G, Paice JA, Wallace MS, Strassels SA, Farrar JT. Revised American Pain Society Patient Outcome Questionnaire (APS-POQ-R) for quality improvement of pain management in hospitalized adults: preliminary psychometric evaluation. J Pain. 2010;11(11):1172–86.
Botti M, Khaw D, Jorgensen EB, Rasmussen B, Hunter S, Redley B. Cross-cultural examination of the structure of the Revised American Pain Society Patient Outcome Questionnaire (APS-POQ-R). J Pain. 2015;16(8):727–40.
Fang H, Liang J, Hong Z, Sugiyama K, Nozaki T, Kobayashi S, Sameshima T, Namba H, Asakawa T. Psychometric evaluation of the Chinese version of the revised American Pain Society Patient Outcome Questionnaire concerning pain management in Chinese orthopedic patients. PLoS One. 2017;12(5):e0178268.
Zoega S, Ward S, Gunnarsdottir S. Evaluating the quality of pain management in a hospital setting: testing the psychometric properties of the Icelandic version of the revised American Pain Society patient outcome questionnaire. Pain Manag Nurs. 2014;15(1):143–55.
Idvall E, Hamrin E, Unosson M. Development of an instrument to measure strategic and clinical quality indicators in postoperative pain management. J Adv Nurs. 2002;37(6):532–40.
Byrne BM. Factor analytic models: viewing the structure of an assessment instrument from three perspectives. J Pers Assess. 2005;85(1):17–32.
Unruh AM. Gender variations in clinical pain experience. Pain. 1996;65(2–3):123–67.
Paller CJ, Campbell CM, Edwards RR, Dobs AS. Sex-based differences in pain perception and treatment. Pain Med. 2009;10(2):289–99.
McArdle JJ. Current directions in structural factor analysis. Curr Dir Psychol Sci. 1996;5(1):11–8.
Collins LM. Research design and methods. In: Birren JE, editor. Encyclopedia of gerontology. 2nd ed. New York: Elsevier; 2007. p. 433–42.
Browne MW, Cudeck R. Alternative ways of assessing model fit. Sociol Methods Res. 1992;21(2):230–58.
Hu LT, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Modeling. 1999;6(1):1–55.
Kline RB. Methodology in the social sciences. Principles and practice of structural equation modeling. 2nd ed. New York: Guilford Press; 2005.
Knapp TR, Brown JK. Ten measurement commandments that often should be broken. Res Nurs Health. 1995;18(5):465–9.
Murno BH, Page EB. Statistical Methods for Health Care Research (2nd edition). Philadelphia: Lippincott Williams & Wilkins; 1993 p. 254.
von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP, Initiative S. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61(4):344–9.
Gordon DB, Dahl JL, Miaskowski C, McCarberg B, Todd KH, Paice JA, Lipman AG, Bookbinder M, Sanders SH, Turk DC, Carr DB. American pain society recommendations for improving the quality of acute and cancer pain management: American Pain Society Quality of Care Task Force. Arch Intern Med. 2005;165(14):1574–80.
Polit DF, Beck CT. Nursing research: principles and methods. Philadelphia: Lippincott Williams & Wilkins; 2004.
Reuben DB, Jennings LA. Putting goal-oriented patient care into practice. J Am Geriatr Soc. 2019;67(7):1342–4.
McTier LJ, Botti M, Duke M. Patient participation in quality pain management during an acute care admission. Clin J Pain. 2014;30(4):316–23.
Ward SE, Gordon DB. Patient satisfaction and pain severity as outcomes in pain management: a longitudinal view of one setting’s experience. J Pain Symptom Manage. 1996;11(4):242–51.
We sincerely thank all the patients who participated in this study.
This study was funded by Bantuan Penyelidikan Kecil (BKP; Small Research Aid) Grant of the University of Malaya (Grant number: BK031-2016) and International Medical University Internal Grants (Grant numbers: BP-I-01/14(38)2017 and BP-I-01/14(40)2017). The funding bodies do not participate in the design of the study, collection, analysis, and interpretation of data, and in writing the manuscript.
Ethics approval and consent to participate
The study protocol was approved by the UMMC Medical Research and Ethics Committee (Approval number: 20161224660), and the International Medical University (IMU) Joint Committee on Research and Ethics (Approval number: IG544), Kuala Lumpur, Malaysia. Written consent from patients was obtained before their participation.
Consent for publication
No conflict of interest from all authors.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Chaw, S.H., Lo, Y.L., Lee, J.Y. et al. Evaluate construct validity of the Revised American Pain Society Patient Outcome Questionnaire in gynecological postoperative patients using confirmatory factor analysis. BMC Anesthesiol 21, 20 (2021). https://doi.org/10.1186/s12871-020-01229-x
- Construct validity
- Factor analysis
- Pain management
- Patient management
- Postoperative pain