- Research article
- Open Access
Assignment of pre-event ASA physical status classification by pre-hospital physicians: a prospective inter-rater reliability study
BMC Anesthesiology volume 20, Article number: 167 (2020)
Individualized treatment is a common principle in hospitals. Treatment decisions are made based on the patient’s condition, including comorbidities. This principle is equally relevant out-of-hospital. Furthermore, comorbidity is an important risk-adjustment factor when evaluating pre-hospital interventions and may aid therapeutic decisions and triage. The American Society of Anesthesiologists Physical Status (ASA-PS) classification system is included in templates for reporting data in physician-staffed pre-hospital emergency medical services (p-EMS) but whether an adequate full pre-event ASA-PS can be assessed by pre-hospital physicians remains unknown. We aimed to explore whether pre-hospital physicians can score an adequate pre-event ASA-PS with the information available on-scene.
The study was an inter-rater reliability study consisting of two steps. Pre-event ASA-PS scores made by pre- and in-hospital physicians were compared. Pre-hospital physicians did not have access to patient records and scores were based on information obtainable on-scene. In-hospital physicians used the complete patient record (Step 1). To assess inter-rater reliability between pre- and in-hospital physicians when given equal amounts of information, pre-hospital physicians also assigned pre-event ASA-PS for 20 of the included patients by using the complete patient records (Step 2). Inter-rater reliability was analyzed using quadratic weighted Cohen’s kappa (κw).
For most scores (82%) inter-rater reliability between pre-and in-hospital physicians were moderate to substantial (κw 0,47-0,89). Inter-rater reliability was higher among the in-hospital physicians (κw 0,77 to 0.85). When all physicians had access to the same information, κw increased (κw 0,65 to 0,93).
Pre-hospital physicians can score an adequate pre-event ASA-PS on-scene for most patients. To further increase inter-rater reliability, we recommend access to the full patient journal on-scene. We recommend application of the full ASA-PS classification system for reporting of comorbidity in p-EMS.
Tailored treatment through adapted choice of therapy, medication and monitoring to each patient is a common principle in hospitals [1,2,3]. In all parts of critical care, decisions are made based on the patient’s condition, including the patient’s comorbidities [1, 2, 4]. Decisions of dose adjusted medication and volume loading before anesthesia are common examples of individualized adaptions in the operating room . Pre-hospital critical care is a continuum, and pre-hospital management is often a part of the patient’s course [5, 6]. As such, stratification on comorbidity, and individualized treatment, is equally relevant and valid for pre-hospital patients. In line with this principle, the patient’s health status before the acute event should be accounted for in triage on-scene and to determine threshold for, and timing of interventions and physiological targets [7, 8].
Risk adjustments allows for better judgement about the effectiveness and quality of alternative therapies . Comorbidity is an important risk adjustment factor when evaluating pre-hospital interventions [9, 10]. In general, there is an agreement that outcome after trauma is influenced by the patient’s physical state before the trauma occurs . Thus, to include a comorbidity measure is a prerequisite for comparisons and improves the precision of outcome prediction for trauma patients [8, 9, 12]. However, to obtain information on comorbidity from in-hospital records may be challenging for pre-hospital services due to logistics and legal issues of access and other strategies for obtaining this information should be explored.
Several methods for reporting comorbidities in pre-hospital emergency medical services (p-EMS) exists [8, 9, 13]. The American Society of Anesthesiologists Physical Scale (ASA-PS) classification system is used globally by anesthesiologists and classifies the preoperative physical health condition in patients before anesthesia and surgery. ASA-PS was originally designed to allow for statistical analyses of outcomes and to standardize terminology [14, 15], not to predict perioperative risk , but research has shown that the ASA-PS correlates well with overall surgical mortality . Although the reliability of ASA-PS may be discussed, the scale is widely accepted as a tool to decide pre-operative health status . The use of ASA-PS has expanded to the pre- and in-hospital critical care environment and pre-event ASA-PS, which is ASA-PS before the present injury or illness,  describes the inherent physiological state of a patient before an event. Pre-event ASA-PS is shown to be an independent predictor of mortality after trauma  and is included in templates for reporting of comorbidity in p-EMS and trauma [18, 19]. We therefore used pre-event ASA-PS as a comorbidity measure for the present study.
Ideally, pre-hospital services should have access to the full patient journal on-scene. Reality is however different and access to the full patient journal tends to be restricted for most pre-hospital services on-scene. P-EMS services must thus commonly base their decisions on the more limited amount of data and observations obtainable on-scene than for in-hospital physicians. Obtaining the complete medical history from seriously ill or injured patients on-scene is considered unfeasible, and reporting a dichotomized pre-event ASA-PS (pre-event ASA-PS 1 or pre-event ASA-PS > 1) is thus often recommended . This simplification of the scale provides a very rough measure of comorbidity with low clinical discriminatory capabilities. Whether an adequate full pre-event ASA-PS can be assessed by pre-hospital physicians based only on the limited information generally available on-scene has not been explored and remains unknown. If scores between pre-and in-hospital physicians do not differ more than between in-hospital physicians, then the pre-hospital scores are just as “correct” as the in-hospital scores and can be used accordingly.
The aim of the present study was to explore whether it is possible for pre-hospital physicians to score an adequate pre-event ASA-PS already while on-scene.
Prospective observational inter-rater reliability study. We assessed the degree of agreement among two raters using the ASA-PS scale under different circumstances to decide whether different access to information influenced the scores. All patients admitted by p-EMS to two Norwegian hospitals during a period of three-months (Stavanger University Hospital 19 Aug – 18 Nov 2016 and St. Olav University Hospital 1 Feb – 30 Apr 2017) were included. Following the inclusion periods, in-hospital physicians scored all included patients (Step 1). Data collection for the second part of the study (Step 2) was finished 21 Mar 2018. All Norwegian p-EMS services are staffed with anesthesiologists and respond to all types of emergency conditions, search and rescue missions and inter-hospital transfers.
We used the pre-event ASA-PS to assess comorbidity. The pre-event ASA-PS does not take the present event into account and describes the physiological state of the patient before an event [8, 11, 21]. The ASA-PS provides a global, subjective index of a patient’s overall health status, and pre-existing medical conditions are categorized on a scale of increasing medical severity (ASA-PS 1–5) .
Step 1. Inter-rater reliability study of pre- versus in-hospital scores
Pre-hospital physicians assigned a pre-event ASA-PS score on-scene based on information available out-of-hospital only. The pre-hospital physicians did not have access to the full patient records. If the physician was unable to decide on a pre-event ASA-PS score on-scene, the score was kept unassigned and the main reason declared. After the three-month inclusion period, three in-hospital anesthesiologists at each of the two sites were given access to full patient records for all included patients at each site. Blinded from the pre-event ASA-PS score allocated by p-EMS each in-hospital physician used this information to assign pre-event ASA-PS scores for the included patients. No specific training for ASA-PS scoring was provided.
Step 2. Inter-rater reliability with equal access to data
Because p-EMS generally do not have access to the full patient journal comparing pre-hospital on-scene scores with in-hospital scores is an asymmetric comparison (as in-hospital physicians have access to more information). We thus did not expect perfect agreement between pre- and in-hospital raters. To assess agreement of pre-event ASA-PS scores when pre- and in-hospital physicians had access to equal data, 20 patients were selected by an on-line randomizer and re-scored by the pre-hospital physicians when given access to complete patient records. The rationale behind this was to assess whether an observed difference in scoring was due to different physicians (pre- versus in-hospital) or different data availability.
We were unable to identify any studies in which pre-event ASA-PS was scored in a real-time pre-hospital setting. Without prior empirical information on the variation of the phenomenon under study we were consequently unable to perform sample size calculations [22, 23]. Statistical rules of thumb for sample size varies in the literature and sample sizes from 10 to 50 is reported . Combining existing advice, we chose to included 20 patients per physician to evaluate inter-rater reliability . If no agreement between pre- and in-hospital physicians for 20 patients could be established, we considered the pre-hospital scores to be irrelevant.
Patients and physicians were anonymized prior to further statistical analyses.
Guidelines for Reporting Reliability and Agreement Studies (GRRAS) was used .
ASA-PS is an ordinal scale and agreement between two ASA-PS measures on the same individual was thus assessed using quadratic weighted Cohen’s Kappa (κw); a modification of Cohen’s Kappa that also accounts for the degree of disagreement between raters . κw is a number between 0 and 1. κw < 0.10 indicates no inter-rater reliability, while 0.11–0.40 indicates slight, 0.41–0.60 indicates fair, 0.61–0.80 indicates moderate and 0.8–1.0 indicates substantial inter-rater reliability .
If two measurement methods are to be considered similar their results should be indistinguishable from one another . Using κw values between pre- and in-hospital physicians as a measure of agreement, we performed minimax hierarchical agglomerative clustering; a method for exploring the inner agreement structure of a dataset . The result from this clustering process is presented visually as dendrograms. Such dendrograms look like up-side-down trees, grouping elements that agree the most near the bottom of the graph, with decreasing agreement (i.e. inter-rater reliability) the higher on the graph. This approach allowed us to visually explore whether the agreement between pre-and in-hospital physicians were indeed indistinguishable from one another. The overall mean agreement  for all pre- versus in-hospital physicians was also calculated. Data were analyzed using IBM SPSS statistics version 22 and R 3.1.0.
Pre-event ASA-PS was registered for a total of 312 patients. We excluded four patients admitted to non-participating hospitals and three patients without identifiable patient records. One physician scored only four patients, three with pre-event ASA-PS 3 and one that could not be scored. This did not allow for κw calculations, as scores were identical, and this physician and corresponding patients were thus excluded. In total 301 patients were available for further statistical analysis.
Pre-hospital physicians scored a median (range) of 21 (5–40) patients. Five patients (2%) could not be scored on-scene (four were unconscious and one was not able to communicate).
The distribution of ASA-PS scores between pre- and in-hospital physicians are presented in Table 1.
κw values for pre-event ASA-PS scores assigned by pre-hospital physicians on-scene, and subsequent scores based on complete patient records by in-hospital physicians are presented in Fig. 1.
κw values ranged from 0.77 to 0.85 among the three in-hospital physicians, and from 0.47 to 0.89 when comparing the pre- to in-hospital physicians. The mean kappa values were 0,67 (PDocs Stavanger), 0,78 (IDocs Stavanger), 0,75 (PDocs Trondheim) and 0,84 (IDocs Trondheim). For most scores (82%) inter-rater reliability between pre-and in-hospital physicians were moderate to substantial (κw > 0.61).
The mean agreement between all pre-hospital physicians and each of the three in-hospital physicians is generally high. However, the three in-hospital physicians tend to agree more with one another than they agree with the pre-hospital physicians. This is demonstrated in Fig. 2.
When pre- and in-hospital physicians scored the same 20 patients with equal access to information, the agreement was strengthened. The difference in inter-rater reliability between the pre- and in-hospital physicians was much smaller, with κw values ranging from 0.65 to 0.93, indicating moderate to substantial agreement. Corresponding dendrograms for the two sites demonstrate that scores from pre- and in-hospital physicians do not cluster but remain largely indistinguishable from one another (Fig. 3).
The present study is a study of ASA-PS scoring in real life situations. As pre-hospital physicians did not have access to the full patient journal (Step 1), perfect agreement in ASA-PS scoring between pre-and in-hospital physicians was not to be expected. When comparing pre- and in-hospital pre-event ASA-PS scores, agreement was generally high ranging from fair to substantial. Most scores (82%) demonstrated moderate (64%) to substantial (18%) agreement, indicating that pre-hospital physicians can obtain sufficient data on-scene to score an adequate pre-event ASA-PS for most patients. Because the total number of pre-hospital scores are high, the impact of uncertainty in the scores, represented by broad 95% confidence intervals in Fig. 1, is reduced.
When pre- and in-hospital physicians scored pre-event ASA-PS on the same patients with access to complete patient records, agreement improved and ranged from moderate (52%) to substantial (48%). This indicates that ASA-PS scores from pre- and in-hospital physicians are indistinguishable from one another when they have equal data access (Fig. 3.). Accordingly, observed differences in pre-event ASA-PS scores in the first part of the study may be attributed to differences in data availability and time pressure on-scene rather than to factors related to individual physicians.
Comorbidity is an important risk-adjustment factor when evaluating pre-hospital interventions and the effect of p-EMS [9, 10]. Additionally, adjustment for comorbidity significantly increase the predictive accuracy of trauma outcome prediction models [9, 12, 31, 32]. The inherent nature of p-EMS favors a method for reporting comorbidities that is both readily available and time effective. ASA-PS is a well-known physical health condition scale, globally applied by anesthesiologists and surgeons, supporting the notion that pre-event ASA-PS may be advantageous for reporting comorbidity in p-EMS. However, studies have found substantial inter-observer variation [21, 33]. Most of these studies are hypothetical case scenarios designed by researchers [8, 16, 21]. In the present study we found that the agreement between pre- and in-hospital scores is acceptable for most patients and argue that pre-event pre-hospital ASA-PS should be applied for documentation of comorbidity in p-EMS.
Obtaining complete medical history from seriously ill patients on-scene is considered unfeasible. Accordingly, a dichotomized pre-event ASA-PS is often reported . This is a very rough measure of comorbidity with low clinical discriminatory ability and will not distinguish between mild and severe systemic disease. Our results indicate that p-EMS can assign an adequate full-scale pre-event ASA-PS score already on-scene.
Significantly less accuracy of assigning ASA-PS is reported for non-anesthesiologists compared to anesthesiologists, possibly limiting the validity of pre-hospital pre-event ASA-PS scores to anesthesiologist-staffed services . Standardized education and encouraged use may decrease variability for less proficient users . Knowledge of comorbidity is relevant for all emergency medical services to aid decision-making and to target the treatment. Reliability of pre-event ASA-PS scored by paramedics is unknown and should be subject for further research. Precise definitions of each ASA-PS class, along with training for use, may improve reliability and usability for all users.
Although the physicians in the present study did not have access to patient records only 2% of the patients could not be scored on-scene, all of which had impaired consciousness. These patients remain a challenge for p-EMS regarding comorbidity assessment. Access to patient records in p-EMS may increase feasibility and precision of pre-event ASA-PS scores and systems for field data access should be available. Summary care records (SCRs) are electronic records of important patient information available for authorized health care staff involved in patient care . The prevalence of summary care records (SCRs) is increasing . SCRs may provide timely and relevant patient information regardless of regional affiliation. Whether access to SCRs will increase reliability of pre-event ASA-PS scores on-scene remains unknown.
The study was performed in a highly specialized anesthesiologist-staffed system and the results may not be transferable to other p-EMS. When number of assigned scores is low, conclusions may be inaccurate. Patients who died prior to hospital arrival were excluded. These patients are among the most severely sick or injured patients and may have a substantial comorbidity burden. Omitting these patients may overestimate the rate of agreement in this study.
For an anesthesiologist-staffed EMS covering a mixed patient population, an adequate pre-event ASA-PS can be assigned on-scene. When data access was equal, pre-event ASA-PS scores by pre- and in-hospital physicians were indistinguishable from each other. When pre-event ASA-PS was scored on-scene with restricted data access, inter-rater reliability was lower, but acceptable. We recommend application of the full pre-event ASA-PS classification system for documentation of comorbidity in p-EMS.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
The American Society of Anesthesiologists Physical Status
Physician-staffed pre-hospital emergency medical services
Guidelines for Reporting Reliability and Agreement Studies
- κw :
Quadratic weighted Cohen’s Kappa
Summary care records
Garrison HG, Maio RF, Spaite DW, Desmond JS, Gregor MA, O'Malley PJ, Stiell IG, Cayten CG, Chew JL Jr, Mackenzie EJ, Miller DR. Emergency medical services outcomes project III (EMSOP III): the role of risk adjustment in out-of-hospital outcomes research. Ann Emerg Med. 2002;40:79–88.
Keim SM, Spaite DW, Maio RF, Garrison HG, Desmond JS, Gregor MA, O'Malley PJ, Stiell IG, Cayten CG, Chew JL Jr, et al. Risk adjustment and outcome measures for out-of-hospital respiratory distress. Acad Emerg Med. 2004;11:1074–81.
Van Gelder IC, Hobbelt AH, Marcos EG, Schotten U, Cappato R, Lewalter T, Schwieler J, Rienstra M, Boriani G. Tailored treatment strategies: a new approach for modern management of atrial fibrillation. J Intern Med. 2016;279:457–66.
Miller RD. Miller’s Anestehesia 6th edition. In: Miller RD, editor. Miller’s Anestehsia, vol. 1. Philadelphia: Elsevier Churcuill Livingstone; 2005. p. 1018.
Vincent JL. The continuum of critical care. Crit Care. 2019;23:122.
Ghosh R, Pepe P. The critical care cascade: a systems approach. Curr Opin Crit Care. 2009;15:279–83.
Scalea TM, Simon HM, Duncan AO, Atweh NA, Sclafani SJ, Phillips TF, Shaftan GW. Geriatric blunt multiple trauma: improved survival with early invasive monitoring. J Trauma. 1990;30:129–34 discussion 134-126.
Skaga NO, Eken T, Sovik S, Jones JM, Steen PA. Pre-injury ASA physical status classification is an independent predictor of mortality after trauma. J Trauma. 2007;63:972–8.
Bouamra O, Jacques R, Edwards A, Yates DW, Lawrence T, Jenks T, Woodford M, Lecky F. Prediction modelling for trauma using comorbidity and ‘true’ 30-day outcome. Emerg Med J. 2015;32:933–8.
Ghorbani P, Ringdal KG, Hestnes M, Skaga NO, Eken T, Ekbom A, Strommer L. Comparison of risk-adjusted survival in two Scandinavian level-I trauma centres. Scand J Trauma Resusc Emerg Med. 2016;24:66.
Jones JM, Skaga NO, Sovik S, Lossius HM, Eken T. Norwegian survival prediction model in trauma: modelling effects of anatomic injury, acute physiology, age, and co-morbidity. Acta Anaesthesiol Scand. 2014;58:303–15.
de Munter L, Polinder S, Lansink KW, Cnossen MC, Steyerberg EW, de Jongh MA. Mortality prediction models in the general trauma population: a systematic review. Injury. 2017;48:221–9.
Austin SR, Wong YN, Uzzo RG, Beck JR, Egleston BL. Why summary comorbidity measures such as the Charlson comorbidity index and Elixhauser score work. Med Care. 2015;53:e65–72.
Keats AS. The ASA classification of physical status--a recapitulation. Anesthesiology. 1978;49:233–6.
Saklad M. Grading of patients for surgical procedures. Anesthesiology. 1941;2:281–4.
Sankar A, Johnson SR, Beattie WS, Tait G, Wijeysundera DN. Reliability of the American Society of Anesthesiologists physical status scale in clinical practice. Br J Anaesth. 2014;113:424–32.
ASA Physical Status Classification System. https://www.asahq.org/standards-and-guidelines/asa-physical-status-classification-system. Accessed 1 Jun 2020.
Ringdal KG, Coats TJ, Lefering R, Di Bartolomeo S, Steen PA, Roise O, Handolin L, Lossius HM. The Utstein template for uniform reporting of data following major trauma: a joint revision by SCANTEM, TARN, DGU-TR and RITG. Scand J Trauma Resusc Emerg Med. 2008;16:7.
Tønsager K, Krüger AJ, Ringdal KG, Rehn M. Template for documenting and reporting data in physician-staffed pre-hospital services: a consensus-based update. Scand J Trauma Resusc Emerg Med. 2020;28:25.
Kruger AJ, Lockey D, Kurola J, Di Bartolomeo S, Castren M, Mikkelsen S, Lossius HM. A consensus-based template for documenting and reporting in physician-staffed pre-hospital services. Scand J Trauma Resusc Emerg Med. 2011;19:71.
Ringdal KG, Skaga NO, Steen PA, Hestnes M, Laake P, Jones JM, Lossius HM. Classification of comorbidity in trauma: the reliability of pre-injury ASA physical status classification. Injury. 2013;44:29–35.
Kirby A, Gebski V, Keech AC. Determining the sample size in a clinical trial. Med J Aust. 2002;177:256–7.
Kadam P, Bhalerao S. Sample size calculation. Int J Ayurveda Res. 2010;1:55–7.
Corder GW, Foreman DI. Nonparametric statistics for non-statisticians. Hoboken: Wiley; 2009.
Kottner J, Audige L, Brorson S, Donner A, Gajewski BJ, Hrobjartsson A, Roberts C, Shoukri M, Streiner DL. Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. J Clin Epidemiol. 2011;64:96–106.
Hallgren KA. Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol. 2012;8:23–34.
Shrout PE. Measurement reliability and agreement in psychiatry. Stat Methods Med Res. 1998;7:301–17.
Zou KH, Warfield SK, Bharatha A, Tempany CM, Kaus MR, Haker SJ, Wells WM 3rd, Jolesz FA, Kikinis R. Statistical validation of image segmentation quality based on a spatial overlap index. Acad Radiol. 2004;11:178–89.
Bien J, Tibshirani R. Hierarchical clustering with prototypes via Minimax linkage. J Am Stat Assoc. 2011;106:1075–84.
Fiori S, Tanaka T. An algorithm to compute averages on matrix lie groups. Trans Sig Proc. 2009;57:4734–43.
Bergeron E, Rossignol M, Osler T, Clas D, Lavoie A. Improving the TRISS methodology by restructuring age categories and adding comorbidities. J Trauma. 2004;56:760–7.
Skaga NO, Eken T, Sovik S. Validating performance of TRISS, TARN and NORMIT survival prediction models in a Norwegian trauma population. Acta Anaesthesiol Scand. 2018;62:253–66.
Riley R, Holman C, Fletcher D. Inter-rater reliability of the ASA physical status classification in a sample of anaesthetists in Western Australia. Anaesth Intensive Care. 2014;42:614–8.
Curatolo C, Goldberg A, Maerz D, Lin HM, Shah H, Trinh M. ASA physical status assignment by non-anesthesia providers: do surgeons consistently downgrade the ASA score preoperatively? J Clin Anesth. 2017;38:123–8.
Ihejirika RC, Thakore RV, Sathiyakumar V, Ehrenfeld JM, Obremskey WT, Sethi MK. An assessment of the inter-rater reliability of the ASA physical status score in the orthopaedic trauma population. Injury. 2015;46:542–6.
Jones EW. How summary care records can improve patient safety. Emerg Nurse. 2015;23:20–2.
The authors are grateful to the donors of the Norwegian Air Ambulance Foundation. The authors thank all pre-hospital physicians in Stavanger and Trondheim who collected pre-hospital data and Guro Mæhlum Krüger, Trond Nordseth, Helge Haugland, Katrine Finsnes, Unni Bergland and Linda Rørtveit who collected in-hospital data.
The Norwegian Air Ambulance Foundation funded this project but played no part in study design, data collection, analysis, writing or submitting to publication.
Ethics approval and consent to participate
The Regional Committee for Medical and Health Research Ethics in Western Norway (ID 2016/556) approved the study and ruled out that no formal consent was necessary, thus; they approved exemption of consent for all patients.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Tønsager, K., Rehn, M., Krüger, A.J. et al. Assignment of pre-event ASA physical status classification by pre-hospital physicians: a prospective inter-rater reliability study. BMC Anesthesiol 20, 167 (2020). https://doi.org/10.1186/s12871-020-01083-x
- Critical care
- Emergency medical services
- Pre-hospital emergency care