- Research article
- Open Access
Identifying appropriate outcomes to help evaluate the impact of the Canadian Guideline for Safe and Effective Use of Opioids for Non-Cancer Pain
BMC Anesthesiology volume 20, Article number: 6 (2020)
The Canadian Guideline for Safe and Effective Use of Opioids for Chronic Non-Cancer Pain (COG) was developed in response to increasing rates of opioid-related hospital visits and deaths in Canada, and uncertain benefits of opioids for chronic non-cancer pain (CNCP). Following publication, we developed a list of evaluable outcomes to assess the impact of this guideline on practice and patient outcomes.
A working group at the National Pain Centre at McMaster University used a modified Delphi process to construct a list of clinical and patient outcomes important in assessing the uptake and application of the COG. An advisory group then reviewed this list to determine the relevance and feasibility of each outcome, and identified potential data sources. This feedback was reviewed by the National Faculty for the Guideline, and a National Advisory Group that included the creators of the COG, resulting in the final list of 5 priority outcomes.
Five outcomes were judged clinically important and feasible to measure: 1) Effects of opioids for CNCP on quality of life, 2) Assessment of patient’s risk of addiction before starting opioid therapy, 3) Monitoring patients on opioid therapy for aberrant drug-related behaviour, 4) Mortality rates associated with prescription opioid overdose and 5) Use of treatment agreements with patients before initiating opioid therapy for CNCP. Data sources for these outcomes included patient’s medical charts, e-Opioid Manager, prescription monitoring programs and administrative databases.
Measuring the impact of best practice guidelines is infrequently done. Future research should consider capturing the five outcomes identified in this study to evaluate the impact of the COG in promoting evidence-based use of opioids for CNCP.
The Canadian Guideline for Safe and Effective Use of Opioids for Chronic Non-Cancer Pain  (COG) was initially released in April 2010 and an updated and revised version was published in 2017 . This guideline was developed in response to the concern that Canada is the second highest user of opioids per capita in the world and that the rates of opioid prescribing and opioid-related hospital visits and deaths have been increasing . The COG is officially endorsed by the Federation of Medical Regulatory Authorities of Canada (FMRAC) and all provincial medical colleges, and is freely available from the website of the Michael G. DeGroote National Pain Centre (NPC), McMaster University  and in a multi-layered and interactive format through MAGICapp . Development of the 2010 COG was undertaken by the National Opioid Use Guideline Group (NOUGG)  working under the auspices of FMRAC . Copyright was then transferred to the NPC which assumed responsibility for dissemination and any future update. Its update was then undertaken by the NPC at McMaster University with funding by Health Canada . The NOUGG established a National Faculty that still exists today and whose members have had input into the COG. The National Faculty consists of professionals with expertise in pain management, addiction medicine, primary care, knowledge translation, and epidemiology and patient advocates and representatives, who are further divided into working groups of experts to ensure the ongoing dissemination, evaluation and revision of the COG. One of the working groups, the Guideline Impact Evaluation Working Group (EWG) was established to evaluate of the impact of the COG in Canada. This included establishing important outcomes to measure and determining where and how to obtain necessary data for guideline evaluation.
In this paper, we report the process used to identify key priority outcomes, and data sources for their measurement, for evaluation of the impact of the COG. The primary objective of this study was to identify those key priority outcomes and their associated measures for future evaluation. A cross-country collaboration is needed to evaluate the issue of opioid addiction and diversion in Canada and assess the COG’s impact on practice and patient outcomes. Thus, it is expected that the resulting outcomes and associated measures from this research reported in this paper will provide direction to the evaluation process.
We used Moore’s model for evaluating educational interventions and their outcomes to identify priority outcomes and their associated measures for evaluation . In Moore’s expanded outcomes model of learning outcomes for assessing educational interventions activities, we considered outcomes at the performance level (level 5) or higher. Level 5 evaluates how well participants demonstrate or do what the educational activity intended for them to do in their practice. The next two levels, patient health (level 6) and community health (level 7), evaluate the degree to which the health status of patients improves due to changes in practice behaviour of clinicians and the health status of a community of patients changes due to changes in practice behaviour .
The process to select outcomes took place over the period following the introduction of the 2010 COG from May 2011 through mid-2013 using a modified Delphi process [9, 10]. Figure 1 outlines the steps taken to select 5 priority outcomes. For Step 1, the EWG consulted with the Canadian Hypertension Education Program (CHEP), now known as Hypertension Canada, which has extensive experience in guideline evaluation [11, 12]. Based on Hypertension Canada’s experience, we reviewed established Canadian health related surveys to determine if they included outcome measures relevant to our objective. We found that the outcome measures from these surveys were not applicable to evaluating the COG and, as such, the EWG generated an initial list of 29 relevant outcomes to evaluate the impact of the 2010 COG and its 24 practice recommendations . A Definitions Outcomes Group consisting of 14 physicians, pharmacists, and methodologists, with expertise in pain and addictions management, knowledge translation, and epidemiology categorized these outcomes as either practice or clinical outcomes, and further subdivided them into pain and addiction categories (Table 1). Reponses and data for Steps 2, 3 and 4 were collected using Opinio, online-based survey software . None of the responses could be linked to an individual. For Step 2, each member of the Definitions Outcomes Group selected their top five choices. Outcomes that each member of the Definitions Outcomes Group chose as their top five were selected and used for Step 3a.
For Step 3a, the outcomes that each member of the Definitions Outcomes Group chose as their top five (from Step 2) were then sent to the original NOUGG National Advisory Panel, Research Group, and the National Faculty (n = 74) who were asked to rank their top five outcomes from 1 (first choice) to 5 (fifth choice). The National Faculty, in addition to including professionals with expertise in pain management, addiction medicine, primary care, knowledge translation, and epidemiology, also includes patient advocates and representatives. We developed weighted scores for each outcome as follows: A first choice (ranking of 1) was assigned 5 points; a second choice (ranking of 2) was assigned 4 points; a third choice (ranking of 3) was assigned 3 points; a fourth choice (ranking of 4) was assigned 2 points; and a fifth choice (ranking of 5) was assigned 1 point. These points were multiplied by the number of respondents giving that ranking and the points summed for a weighted score. For example, an outcome ranked first by 11 people, second by 7 people, third by 8 people, fourth by 1 person, and fifth by 5 people received a weighted score of 114 [(5 points × 11 people) + (4 points × 7 people) + (3 points × 8 people) + (2 points × 1 person) + (1 point × 5 people) = 114].
For Step 3b, the Definitions Outcome Group identified potential data sources to measure each of the outcomes from Step 2. This was done concurrently with Step 3a. Each data source was rated on a 5-point scale (a point of 1 meant “least” and a point of 5 meant “most”) based on the following criteria adapted from the Nova Scotia Department of Health and Wellness, Hypertension Indicator Priority Setting Tool :
Feasibility - Data to measure outcome should be easily accessible;
Credibility - Should be valid and reliable;
Comparability - Could be used to compare across geographic areas and across time; and
Understandability - Should be easy to interpret with no ambiguity as to whether performance has improved or deteriorated.
For Step 4, the list of outcomes determined at Step 2 and potential data sources from Step 3b were sent again to the NOUGG National Advisory Panel and Research Group and the National Faculty (n = 74) soliciting their feedback on which outcomes they thought could be easily measured, and how, In addition, they were asked to identify potential collaborators to assess the outcomes that will generated from this study to help evaluate the impact of the COG on practice.
For Step 5, in a face-to face discussion the Definitions Outcome Group agreed to select 5 outcomes as priority for the evaluation of the impact of the COG, based on the following information in its totality:
The weighted scores and associated rank of the outcomes from Step 3a;
The evaluation of the potential data sources for the outcomes based on feasibility, credibility, comparability and understandability from Step 3b;
Solicited feedback from Step 4; and
The level the outcome meets on the Moore’s model for evaluating educational interventions.
Any outcome at or above the performance level (Level 5) of the Moore’s model for evaluating educational interventions deemed by the Definitions Outcome Group was considered. We think it necessary to aim for a conceptually high level on the Moore’s model when evaluating a national clinical guideline as outcomes at or above Level 5 are better able to indicate an impact on clinical practice and patient outcomes.
For Step 1, the EWG generated an initial list of 29 outcomes to evaluate the impact of the 2010 COG and its 24 practice recommendations  (Table 1). For Step 2, 16 of the 29 outcomes were selected by at least one Definitions Outcomes Group member as their top five (Table 2). Table 3 highlights the weighted scores and the rankings from Step 3a. Forty-five out of 74 people responded to this survey for a response rate 60.8%. Weighted scores ranged from 14 to 146. Table 4 highlights the data sources identified to measure each of the 16 outcomes from Step 2 and the ratings for each data source on a 5-point scale, based on feasibility, credibility, comparability and understandability. For Step 4, seven out of 74 people responded and they suggested if possible, to not rely on self-reported data and identified potential collaborators such as Workers Compensation Boards which have records on medication prescriptions for clients, Prescription Monitoring Programs, and the Centre for Effective Practice . It was also mentioned that there is a point of care tool for prescribing opioids called the Opioid Manager  which is available in an electronic format and amenable to data collection.
For Step 5, the Definitions Outcome Group selected the following five outcomes (See Table 5) as priority for the evaluation of the impact of the COG on practice and patient outcomes:
Effects of CNCP and taking opioids for CNCP on quality of life
Assessment of patient’s risk of addiction before starting opioid therapy by prescribers, such as the use of standardized tools.
Monitoring patients on opioid therapy for aberrant drug-related behaviours
Mortality rates associated with prescription opioid overdose
Use of treatment agreements with patients before initiating opioid therapy for CNCP
Table 5 summarizes the main discussion points of the Definitions Outcome Group when they assessed and chose these five priority outcomes to help evaluate the impact of the COG on practice and patient care; including the associated measures and data sources that could be used, the ranking they received from NOUGG members and the National Faculty from Table 3 and the level the outcome was on the Moore’s model .
Our study identified five priority outcomes, and a series of possible associated measures and data sources, to evaluate the impact of the COG. Data to assess four of the five identified priority outcomes (Outcomes 1, 2, 3 and 5) would be best collected directly by using electronic patient health records, and integrated electronic practice tools, such as the Opioid Manager. The Opioid Manager is both a tool and source of data specifically developed to monitor patients with CNCP and on opioids, and captures these outcomes [6, 17, 18]. Medical charts may be another source of data for these outcomes. For Outcome 1, regarding the effects of CNCP and taking opioids for CNCP on quality of life, the Short Form-36 (SF-36) is an example of a validated and reliable measure to assess a patient’s quality of life  and can be used in parallel with the Opioid Manager. This example measure, the SF-36, was not identified as a result of this study’s process for selecting outcomes but rather is provided as good example of a measure that can be used for Outcome 1. For Outcome 3, monitoring patients on opioid therapy for aberrant drug-related behaviour, one could also utilize prescription monitoring programs. Outcome 4, mortality rates associated with prescription opioid overdose, could be informed through administrative databases (e.g. coroner’s data and electronic health records). However, further examination of Outcome 4 is warranted as an opioid overdose can involve multiple types of drugs, such as more than one opioid (e.g. fentanyl and an opiate alkaloid), and alcohol. Thus, assessment of mortality rates associated with opioids requires concurrent study of a person’s opioid and other medication use history prior to an opioid overdose . This includes the characterization of prescribed opioids deemed 1) responsible for the overdose, 2) contributing to the overdose (e.g. multiple opioid types or Central Nervous System depressants identified from coroner’s report), or 3) influential in a patient’s history relating to their overdose (e.g. prescribed in their past), and assessment of their concomitant medications. As mentioned above, this characterization can also be found in a coroner’s report and in a patient’s electronic health records.
Explanation of the findings
We believe this research provides insight into a systematic approach that was and can be used again in the future to validate outcomes for guideline evaluation and evaluation processes for other opioid guidelines and perhaps for other guidelines in general. We undertook this systematic approach with the intent to assess the impact of the COG on practice and patient care. The COG provides a knowledge base which can support consistent education and outcome measures. It is helpful to consider our five priority outcomes in relation to Moore’s expanded outcomes model of learning outcomes for assessing education interventions ; one of the outcomes (Outcome 4) is at Level 7 (community health outcome), Outcome 1 is at Level 6 (Individual patient health outcomes), and Outcomes 2, 3, and 5, are at Level 5 (Clinicians’ application of knowledge in practice setting or competency).
Currently, there is a lack of studies evaluating the impact of the COG and medical guidelines in general on clinical practice and patient outcomes in Canada [21, 22]. In relation to other efforts and studies to evaluate the impact of guidelines or recommendations on clinical practice, much work has been done by Hypertension Canada to monitor the impact of such efforts to deal with hypertension in Canada in addition to an implementation (knowledge translation) program for such guidelines into primary care [11, 23]. Their initial key outcome was to improve the treatment and control of hypertension in Canada by creating and implementing recommendations. In 2003, they then set up a task force of experts in the field to assess their efforts to improve the treatment and control of hypertension . Their task force, similar to ours, identified a number of areas or outcomes for surveillance in relation to hypertension. They used nationally administered surveys through Statistics Canada as a tool for surveillance. In the future, it may be possible for us to also work with Statistics Canada to include questions addressing outcome 1, on the effects of CNCP and taking opioids for CNCP on quality of life and other outcomes where feasible.
There is little in the way of formal guidance available in evaluating the impact of clinical practice guidelines. Thus, the working group found it necessary to seek guidance from several sources, and convene several groups of advisors, in order to assist in the de novo development of outcomes relevant to the clinical areas addressed by the guideline. We believe that this potential limitation was mitigated by the strength and breadth of expertise of the advisors involved. For instance, the Evaluation Working Group, and the National Pain Centre, which is the sponsoring body for the National Faculty, includes practitioners in the fields of pain and addiction, research scientists, clinicians, pharmacists, physicians and nurses for multi-disciplinary input. The Definitions Outcome Group included scientists with experience in population health research who informed the feasibility of proposed measures, and individuals who had published in support of greater restriction of opioid use as well as greater availability of opioids for analgesia, representing the breadth of clinical problems faced. Additionally, the 5 priority outcomes generated from this process were determined after introduction of 2010 COG but prior to the 2017 update. These priority outcomes are still relevant to assess as most still relate to the updated guideline recommendations and are based on upon a rigorous process that solicited the view of professionals with expertise in pain management, addiction, knowledge translation, epidemiology and patient advocacy, who deemed these outcomes worth evaluating in regards to CNCP management and proper opioid use. However, it is important to note that both the 2010 COG  and the most recent 2017 COG  were unable to make a strong recommendation or provide strong evidence respectively about the use of treatment agreements, even though it was found in our study that such an outcome would be of importance to evaluate. This suggests it is very likely, in the future, that this outcome would still come up as something important to evaluate as it did in our study despite both the 2010 and 2017 COG stating that such a tool may be helpful to use. This outcome was evidently in the minds of who were surveyed in this study and the modified Delphi process we undertook drew out this outcome.
We identified five priority outcomes, and a series of possible associated measures and data sources to target the evaluation of the guideline’s impact on practice and patient care. Our goal is to make these five priority outcomes widely known to investigators and organizations for consideration in the design of future research studies or funding calls. We welcome collaboration from other groups and individuals as we move forward on evaluating the five priority outcomes we have identified. We also welcome input from and collaboration with groups that may have practical ideas about measuring other outcomes, or have existing data that could contribute to the COG’s evaluation. Our approach is to be collaborative and comprehensive with evaluating the impact of COG on practice.
Availability of data and materials
Canadian Hypertension Education Program
Canadian Guideline for Safe and Effective Use of Opioids for Chronic Non-Cancer Pain
Evaluation Working Group
Federation of Medical Regulatory Authourities of Canada
Opioid Use Guideline Group
National Pain Centre
Canadian Guideline for Safe and Effective Use of Opioids for Chronic Non-Cancer Pain. [ http://nationalpaincentre.mcmaster.ca/opioid/].
Busse JW, Craigie S, Juurlink DN, Buckley DN, Wang L, Couban RJ, Agoritsas T, Akl EA, Carrasco-Labra A, Cooper L. Guideline for opioid therapy and chronic noncancer pain. Can Med Assoc J. 2017;189(18):E659–66.
National Pain Centre, Guidelines [http://nationalpaincentre.mcmaster.ca/guidelines.html].
The 2017 Canadian Guideline for Opioids for Chronic Non-Cancer Pain [http://nationalpaincentre.mcmaster.ca/guidelines.html].
Canadian Guideline for Safe and Effective Use of Opioids for Chronic Non-Cancer Pain. Appendix A-1 [http://nationalpaincentre.mcmaster.ca/opioid/cgop_a_app_a01.html].
Furlan AD, Reardon R, Weppler C. National Opioid use Guideline G: opioids for chronic noncancer pain: a new Canadian practice guideline. CMAJ. 2010;182(9):923–30.
Laine C, Taichman DB, Mulrow C. Trustworthy clinical guidelines. Ann Intern Med. 2011;154(11):774–5.
Moore DE Jr, Green JS, Gallis HA. Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities. J Contin Educ Heal Prof. 2009;29(1):1–15.
Penciner R, Langhan T, Lee R, Mcewen J, Woods RA, Bandiera G. Using a Delphi process to establish consensus on emergency medicine clerkship competencies. Medical teacher. 2011;33(6):e333–9.
Uphoff EP, Wennekes L, Punt CJ, Grol RP, Wollersheim HC, Hermens RP, Ottevanger PB. Development of generic quality indicators for patient-centered cancer care by using a RAND modified Delphi method. Cancer Nurs. 2012;35(1):29–37.
Campbell N, Onysko J. Canadian hypertension education P, outcomes research task F: the outcomes research task Force and the Canadian hypertension education program. Can J Cardiol. 2006;22(7):556–8.
Hypertension Canada [https://hypertension.ca/].
Canadian Guideline for Safe and Effective Use of Opioids for Chronic Non-Cancer Pain - Part B: Recommendations for Practice [http://nationalpaincentre.mcmaster.ca/documents/opioid_guideline_part_b_v5_6.pdf].
Hosted survey solutions [http://objectplanet.com/opinio/hosted-survey-solution.html].
Nova Scota Department of Health and Wellness. Hypertension Indicator Priority Setting Tool. Halifax, Nova Scotia: The Department of Health and Wellness; 2012.
Centre for Effective Practice [https://effectivepractice.org/].
Opioid Manager [https://www.opioidmanager.com/].
Opioid Manager [http://nationalpaincentre.mcmaster.ca/opioidmanager/].
Elliott TE, Renier CM, Palcher JA. Chronic pain, depression, and quality of life: correlations and predictive value of the SF-36. Pain Med. 2003;4(4):331–9.
Force CPT. Chronic pain in Canada: laying the Foundation for Action. Canada: Health Canada; 2019.
Barth KS, Guille C, McCauley J, Brady KT. Targeting practitioners: a review of guidelines, training, and policy in pain management. Drug Alcohol Depend. 2017;173:S22–30.
Chatterjee A, Bhattacharyya O, Persaud N. How can Canadian guideline recommendations be tested? Cmaj. 2013;185(6):465–7.
Bolli P, Campbell NR. Do recommendations for the management of hypertension improve cardiovascular outcome? The canadian experience. Int J Hypertens. 2011;2011:410754.
This work was supported by the Michael G. DeGroote National Pain Centre (NPC), McMaster University, Hamilton. The Michael G. DeGroote NPC is funded by an endowment from Michael G. DeGroote, a philanthropist, and is administered by McMaster University, independent of any funding contingencies. All costs related to this manuscript, including those related to the design of the study and collection, analysis, and interpretation of data and in writing the manuscript, were borne by the NPC.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Allen, M., Sproule, B., MacDougall, P. et al. Identifying appropriate outcomes to help evaluate the impact of the Canadian Guideline for Safe and Effective Use of Opioids for Non-Cancer Pain. BMC Anesthesiol 20, 6 (2020). https://doi.org/10.1186/s12871-020-0930-4
- Outcomes to evaluate guideline impact
- Modified Delphi process
- National Pain Centre
- Chronic non-Cancer pain