Development of the questionnaire
Interviews
On the basis of the extant literature, a semi-structured interview guide was developed focusing on the following three main areas: 1) perceptions of the conditions which could/should induce considerations about withholding or withdrawing therapy in the ICU; 2) the challenges which were experienced regarding withholding or withdrawing therapy; and 3) the perception of what characterizes “good” and “poor” decision processes regarding withdrawal of therapy. In order to ensure that all issues were identified which were assessed as important by the participants, the participants were finally asked: “Are there other important areas regarding end-of-life decisions we have not talked about?”
Four focus group interviews, two with nurses and two with intensivists, were conducted (between 4 and 6 participants in each group; 21 in total). The participants were randomly selected between all staff from 2 ICUs from different hospitals in the study region (ICU A and B), with staff experience taken into account, so all groups consisted of staff with short, medium, and long experiences from the ICU. The nurses were all females, whereas there were 4 female and 6 male intensivists. The intensivists included anaesthesiologists with the ICU as the main workplace and anaesthesiologists who only worked in the ICU on shifts.
From the specialities with the largest percentage of patients admitted to the ICU, primary physicians providing care for patients in the hospital prior to ICU transfer were included. For each ICU, four primary physicians were approached. The physicians were identified by the head of department, the senior secretary, or by the ICU. All primary physicians (4 females and 4 males) agreed to participate in a semi-structured, individual interview based on the same interview guide.
Both focus groups and individual interviews were audio-taped, verbally-transcribed, analysed, and the main themes were extracted [11, 12]. The themes included collaborative issues, changing of withholding decisions which were already made, and unnecessary postponement of withdrawing therapy decisions. All three groups of staff agreed that the main challenge was the process of making a decision of withholding or withdrawing therapy, whereas care given to patients and relatives in whom therapy was withdrawn was found to be easy and good.
Questionnaire
Based on the extant literature, focus groups, and individual interviews, a questionnaire was developed in accordance with the recommendations for questionnaire design by A. Bowling [13]. In the initial process, questions were formulated, assessed, refined, or rejected in collaboration with a team of teachers in questionnaire methodology, quality managers, and intensive care nurses and physicians.
The pilot questionnaire consisted of 39 questions. Apart from questions on background characteristics, the questionnaire had 4 questions regarding reasons for withholding or withdrawing therapy, 2 questions on the process of admitting patients to the ICU, 12 questions on the decision making process regarding withholding or withdrawing therapy, 7 questions on collaboration, 3 questions on documentation, and 4 questions on care for patients and relatives in whom therapy had been withdrawn.
Although the questions were grouped by main themes, the questions were all single-item questions (each question covering a specific area with no automatic correlation expected with other question). Therefore, none of the questions permitted sum-scores [14].
The questions consisted of two types. The first type of question had responses on two different, four- or five-point Likert scales (“very satisfactory”, ”satisfactory”, ”less satisfactory”, and ”unsatisfactory”, or “always”, “often”, “sometimes”, “rarely”, and “never”, both with a “don’t know” option). The second type of question was mostly used to explain reasons for a type 1 question (“If you experience that this is happening, what are the reasons in your opinion?”) or to state recommendations, and these questions had multiple response options. For most of the questions, comments could be added.
The questionnaires were identical for the three staff groups, with the exception of a few questions about background characteristics, and a question for the primary physicians about how often they had patients they wanted to transfer to the ICU, but who were refused by the intensivists.
Validation process
The validation process consisted of the following 3 phases: a qualitative test with 17 participants from ICU A and B (from different hospitals within the study region); a quantitative pilot test with 60 participants from ICU I and II (from different hospitals outside the study region); and a survey with 776 participants from 10 ICUs (from all 7 hospitals in the study region except for primary physicians who came from only 2 of the ICUs) (Figure 1).
The aim of the validation process was to test for:
-
1.
Face validity (does the questionnaire “look like" it is going to measure what it is supposed to measure?) and content validity (do the questions reflect areas that are essential/useful to clarify the purpose of the study?) [15]. Both face and content validity were examined by interviewing pilot study participants after they had responded to the questionnaire. The participants were asked about their general perception of the questionnaire, how they had understood the individual questions, if areas of importance in regard to the subject of the questionnaire were missing, and if any of the questions in their opinion were irrelevant.
-
2.
Reproducibility, which includes reliability and agreement [15]. Reliability is here understood as the questionnaires ability to distinguish between different respondents. If all survey participants respond identically to the individual items, reliability will be low. Reliability is here examined by looking at the distribution of responses. Agreement is understood as consistency in a test-retest, and it is in this study examined with a weighted kappa analysis of responses provided by the same participants within a two week period.
-
3.
Responsiveness, which means the questionnaires ability to detect improvements [15]. As no intervention was tested in the pilot study, known groups’ validity was used as a surrogate concept. Whereas ICU I had no special focus on end-of-life practices, ICU II had focused intensively on end-of-life issues and had developed and implemented guidelines for withdrawal of therapy. Known groups’ validity was tested based on the hypothesis that ICU II would be more satisfied with collaboration, less frequently experience withdrawing of therapy being unnecessarily postponed, and find care of dying patients more satisfactory compared to ICU I. We hypothesised that if the instrument’s ability to detect a difference between the two ICUs was high, the instrument would presumably also be able detect improvements within the individual ICU.
-
4.
Floor/ceiling effect. A high floor and/or ceiling effect may reduce reliability and prevent the ability of the instrument to detect improvement because substantial samples of respondents have already reached the lowest/highest score [15].
-
5.
Missing data. The data was analysed to assess whether or not specific questions had a high percentage of missing data or the percentage of missing data increased in the last questions, indicating that the questionnaire was too long.
Survey
Qualitative pilot test
The questionnaire was first tested qualitatively for face and content validity. The test was conducted with 11 nurses and 5 intensivists from ICU A and B, and 1 primary physician affiliated with ICU A.
Quantitative pilot test
The questionnaire was adjusted based on results from the qualitative pilot study, then tested quantitatively in 2 ICUs from a region not part of the main study. In ICU I, 20 nurses and 10 physicians received the questionnaire twice with a 2-week interval. In ICU II another 20 nurses and 10 physicians received the questionnaire once. In both ICUs, all participants were selected by the charge nurse. The participants were asked to comment on relevance and phrasing of questions.
Main survey
After the pilot study, the questionnaire was used in a survey including 495 nurses and 135 intensivists from 10 ICUs and 146 primary physicians from 2 ICUs [16]. Nurses and intensivists who had been in the ICU for at least 5 months and primary physicians who had attended their patients in the ICU during the last 5 months were eligible for participation in the study. All participants received the questionnaire at their place of employment. Two reminders were sent. The study was conducted in April-June 2010 for nurses and intensivists and in June-August 2010 for primary physicians. Results from the main survey, including a copy of the questionnaire as a supplemental file, are published [16].
Data analysis
All data were double-entered in EpiData (version 3.1), and statistical analyses were performed using STATA 10.1. A chi-square test and Mann–Whitney U test were used to compare background characteristics. Descriptive statistics were used to examine missing data, and the floor and ceiling effect. The weighted kappa and Wilcoxon signed-ranked test were used to test for agreement [17]. Known groups’ validity was tested by a comparison between ICU I and ICU II, using the Mann–Whitney U test. The level of significance was set at a p < 0.05.
Interview data and questionnaire comments were analysed using the meaning condensating method introduced by Kvale [11] and content analysis [12].
Ethics
According to Danish law, the study did not require permission from the Regional Ethics Committee, as confirmed by the Regional Ethics Committee. Permission to obtain and store code lists of staff was granted from The Danish Data Protection Agency (j.o. 2009-41-3189). All Heads of Departments gave permission for their staff to take part in the survey. All participants were informed in writing of the purpose of the study, that participation was voluntary, and that responses were anonymous.
