Email updates

Keep up to date with the latest news and content from CAPMH and BioMed Central.

Open Access Research

Parent experiences questionnaire for outpatient child and adolescent mental health services (PEQ-CAMHS Outpatients): reliability and validity following a national survey

Andrew M Garratt1*, Oyvind A Bjertnaes1, Olaf Holmboe1 and Ketil Hanssen-Bauer23

Author Affiliations

1 Norwegian Knowledge Centre for the Health Services, Oslo, Norway

2 Centre for Child and Adolescent Mental Health, Eastern and Southern Norway, Oslo, Norway

3 Department of Research and Development, Division of Mental Health Services, Akershus University Hospital, Lørenskog, Norway

For all author emails, please log on.

Child and Adolescent Psychiatry and Mental Health 2011, 5:18  doi:10.1186/1753-2000-5-18


The electronic version of this article is the complete one and can be found online at: http://www.capmh.com/content/5/1/18


Received:3 December 2010
Accepted:21 May 2011
Published:21 May 2011

© 2011 Garratt et al; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstact

Background

Development and evaluation of the PEQ-CAMHS Outpatients, a parent completed questionnaire to measure experiences of outpatient child and adolescent mental health services (CAMHS) in Norway.

Methods

Literature review, parent interviews, pre-testing and a national survey of 17,080 parents of children who received care at one of the 86 outpatient CAMHS in Norway in 2006. Telephone interviews were conducted with a random sample of non-respondents. Levels of missing data, factor structure, internal consistency and construct validity were assessed.

Results

7,906 (46.0%) parents or primary caregivers responded to the questionnaire. Low levels of missing data suggest that the PEQ-CAMHS is acceptable. The questionnaire includes three scales supported by the results of factor analysis: relationship with health personnel (8 items), information and participation (4 items), and outcome (3 items). Item-total correlations were all above 0.6 and Cronbach's alpha correlations ranged from 0.88-0.94. The results of comparisons of scale scores with several variables relating to global satisfaction, outcome, cooperation, information, involvement and waiting time support the construct validity of the instrument.

Conclusions

The PEQ-CAMHS Outpatients questionnaire includes important aspects of outpatient CAMHS from the perspective of the parent. It has evidence for data quality, internal consistency and validity and is recommended in surveys of parent experiences of these services. Future research should assess test-retest reliability and further tests of construct validity that include clinical data are recommended.

Background

The widespread acceptance of the importance of the views of service users in the assessment of health care quality has meant that the views of parents are increasingly sought in evaluations of the quality of care of child and adolescent mental health services (CAMHS) [1,2]. Compared with the patient satisfaction literature more generally, [3] there have been relatively few studies of user experiences with CAHMS but there is some evidence for small to moderate levels of correlation between satisfaction and mental health outcomes for children [4-7].

Parents have responsibility for contacting health services on behalf of their child and their involvement in treatment is important for its success [8-10]. Moreover, parent experiences of care might influence their expectations of and involvement in future care [1,11]. Hence, parents can make an important contribution in the evaluation of mental health services quality for children and parent-completed questionnaires have had applications across a wide range of evaluations that include medication [12,13], community-based services [1,14,15], inpatient care [7,11] and health care plans [16].

Reviews of the literature have identified up to 28 instruments and questionnaires designed to measure parent satisfaction with CAMHS [9,17]. These questionnaires typically comprise multidimensional scales that assess different aspects of parent satisfaction or experiences with such services. However, some questionnaires comprise global items while others sum to form a single score that might assess one aspect of parent satisfaction or general satisfaction [18]. Global items have been criticised for not providing adequately detailed information [9,11,19]. Multidimensional instruments provide more detailed information relating to specific aspects of care such as communication and information that can more fully inform quality improvement initiatives.

Parent satisfaction questionnaires have been criticised more generally for not being comprehensive [20] and for lacking psychometric evidence including reliability and validity[7,9,20]. The former criticism relates to content validity, a qualitative consideration as to whether an instrument or questionnaire adequately covers important aspects of parent satisfaction and in sufficient depth. An alternative approach involves asking users to rate their experiences of aspects of health care which may include communication and information provision, [21] and the collection of more objective information relating to whether specific health care events occurred. There is an inbuilt assumption that the aspects of experience covered by such questionnaires are related to user satisfaction and tests of construct validity often involve comparisons with global or single item measures of satisfaction [21]. It is important that users are involved in the development of such questionnaires to ensure that the most relevant aspects of the health care experience are included which lends the questionnaire content validity.

The Parent Experiences Questionnaire for Outpatient Child and Adolescent Mental Health Services (PEQ-CAMHS Outpatients) was developed following a review of the literature that identified shortcomings with existing questionnaires including a lack of content relevance to outpatient clinics in Norway. The questionnaire was designed for application in a national survey of parents or primary caregivers with children under 16 years of age attending mental health services outpatient centres throughout Norway [22]. The aim of the present study was to assess the data quality, internal consistency reliability and validity of the questionnaire.

Methods

Questionnaire development

Questionnaire development and evaluation followed criteria that have been defined as important in patient satisfaction measurement [23]. The development of the questionnaire was based on a review of the literature that included existing parent satisfaction questionnaires, interviews with parents and consultation with an expert group comprising nine health professionals, a representative from a user organisation and researchers. This process was designed to ensure content validity; that the questionnaire addresses important aspects of parent experiences in sufficient detail.

Semi-structured interviews were undertaken with parents of eleven children under 16 years of age that had received care from two outpatient CAMHS in South East Norway. Following the interviews and any necessary revisions, the expert group was consulted. The questionnaire was then piloted by means of a postal questionnaire sent to 271 parents of children under 16 years of age that had attended one outpatient CAMHS in South East Norway. The expert group was consulted again following the pilot survey.

Data collection

The questionnaire was included in national survey of 17,871 parents of children aged under 16 years who had at least one appointment at one of the 86 outpatient centres in Norway between 1 September and 31 December 2006 [22]. Patients were recruited using administrative data from the clinics and there was a maximum of 400 users per clinic who were randomly selected if the number exceeded 400 during the recruitment period. This calculation was based on the number of respondents needed to compare the 86 clinics after taking account of non-response and the need for adjustment of covariates.

Parents were sent a postal questionnaire following the transfer of administrative data from the clinics. The covering letter informed them that the survey related to parent experiences. No instruction was given about which parent should complete the questionnaire and a question was included about who actually responded. Parents gave their informed consent to take part by ticking a box on the front page of the questionnaire which was returned in a reply paid envelope. Up to two reminders were sent to non-respondents at three and six weeks.

Telephone interviews were conducted with a sample of 400 non-respondents following the second reminder; 20 centres were randomly selected from the 86 centres in Norway and 25 patients were randomly selected for each centre. The interviews included four items that were found to explain the largest component of variation in parent experiences following the pilot survey along with background questions. These four questions were summed to form a summary score which had satisfactory item-total correlations and a Cronbach's alpha of 0.84. The two groups of parents were compared using the t-test for the summary score and chi-square and Mann-Whitney U tests for categorical variables and ordinal variables.

The Norwegian Regional Committee for Medical Research Ethics, the Data Inspectorate and the Norwegian Board of Health approved the survey.

Statistical analysis

Items with ten percent or more missing data were considered for removal from the final questionnaire. Principal axis factoring with promax rotation was used to assess the underlying structure of the questionnaire [24]. Factors with eigenvalues above 1.0 were assessed for face validity and factor loadings greater than 0.4 were considered important. Items with important factor loadings on more than one factor were considered for removal. Internal consistency was assessed by item-total correlation and Cronbach's alpha. The former measures the strength of association between an item and the remainder of its scale and it was expected that they would exceed 0.4 which is considered acceptable [25]. The latter assesses the overall correlation between items within a scale. For a scale to be considered sufficiently reliable for use in groups of patients, an alpha value of 0.7 is considered acceptable [25,26].

In the absence of a gold standard measure of parent experiences, construct validity was assessed through comparisons of scale scores with responses to the additional questionnaire items not included in the scales that related to various aspects of care. Following findings from the parent satisfaction literature, [27,28] it was hypothesised that moderate to high levels of correlation in the range 0.5 to 0.7, would be found between scale scores and parents overall satisfaction both in relation to their child's treatment and their own experiences. There is some evidence for a low to moderate association between parent satisfaction and treatment outcomes for mental health services [4-7], and correlations in the range 0.3 to 0.6 were hypothesised for the perceived benefits of treatment accruing to the child and parent. Similar levels of correlation were hypothesised for responses to two items relating to health professionals cooperation, both within the clinic and with other professionals working outside the clinic including general practitioners and school teachers.

The active participation of parents and families in mental health care of children and adolescents leads to better outcomes including parent satisfaction [10,14] and parent satisfaction was found to have low to moderate correlations with a measure of empowerment [10]. Hence, levels of correlation in the range 0.3 to 0.6 were expected between the scales and two items relating to parent understanding of treatment and involvement in dialogue with the clinic. Information giving is also positively related to user satisfaction with health services [3] and it was hypothesised that scale scores would have low to moderate correlations in the range 0.3 to 0.6 with perceptions of the adequacy of information relating to side-effects of their child's medication.

There is evidence that parents are more satisfied with shorter waiting times [14,28] and it was hypothesised that scores would have low levels of correlation in the range 0.2 to 0.3 with how long parents had to wait for treatment for their children. Parent satisfaction ratings have been found to be positively associated with the time spent in treatment with mental health services [19,29,30] and hence it was hypothesised that scores would have low to moderate correlations in the range 0.3 to 0.6 with parent perceptions of the adequacy of the number of hours spent at the clinic, number of clinic visits and the ease with which they were able to make contact with health professionals outside of consultation times when necessary.

Unwanted behaviour by health professionals has been found to be consistently associated with user satisfaction more generally [3] and following previous findings from user satisfaction with mental health services [31], low correlations in the range 0.2 to 0.3 were expected with responses to two items relating to whether health professionals talked down to either the child or their parent.

Results

Questionnaire development

Following the literature review and consultation with the expert group 28 questions were devised that covered accessibility, cooperation, information, overall satisfaction, outcomes and relationship with health personnel. These questions were included in the semi-structured interviews with the parents of eleven children. In general the questions were found to be acceptable and relevant. Some revisions were made following the interviews and consultation with the expert group.

Of the 271 parents sent the pilot questionnaire, 122 (45.0%) responded. The mean age of respondents was 40.83 (sd 6.67) and 93 (76.2%) were the mother. Items had low levels of missing data and factor analysis of 14 items that were relevant to all patients identified three scales of relationship with health personnel, information and participation and outcome. Following consultation with the expert group some small changes were made to the wording of some of the items prior to the main survey and an additional item relating to "how the child functions outside of the family" was included as part of the outcome scale. Hence the PEQ-CAMHS Outpatients questionnaire included 15 items assessing parent experiences with five-point descriptive scales that assess various aspects of parent experiences of relevance to all parents.

Additional items relating to information on medication information and cooperation between health professionals that were not relevant to all parents' included a "not applicable" option and other additional single items covered accessibility, overall outcome, overall satisfaction and unwanted behaviour. Finally, the questionnaire included several sociodemographic questions.

Data collection

Of the 17,871 parents mailed the national survey questionnaire, 7,906 responded; 755 did not have a correct address and 36 had cancelled their CAMHS appointment which gives an adjusted response rate of 46.3%. Table 1 shows their characteristics. The mean age of the children was 11.3 (SD = 3.2) years and 2,946 (37.3%) were female. Of the 400 non-respondents conducted by telephone, 225 (56.3%) agreed to be interviewed. Compared to respondents to the main survey, there were three statistically significant differences that were important. Interviewees had a significantly different number of previous contacts with the clinic with 21.0% of interviewees and 14.4% of respondents reporting more than one previous contact. Interviewees also had a significantly different education level with 25.3% having received higher education compared with 44.3% for respondents. The parent experiences summary scores were not significantly different. However, interviewees had slightly poorer experiences relating to their "viewpoints being taken seriously". There was also a significant difference for the number of clinic visits in the past three months. There were a large number of interviewees that had not visited the outpatient clinic in the past three months which follows from the interviews taking place after the second reminder.

Table 1. Child and parent characteristics for the main survey (n = 7,906) and non-respondents who were telephoneda (n = 225)

Statistical analysis

The levels of missing data and descriptive statistics are shown in Table 2. All items met the criterion for missing data and none were removed. Missing data ranged from 1.5% to 5.6% for items relating to "being met with politeness and respect" and "outcomes associated child's function outside of the family" respectively. These two items also had the highest and lowest scores of 4.35 and 2.39 respectively. The majority of items had mean scores in the range of 3 to 4 on the scale of 1 to 5 where 5 is the best possible experiences of care.

Table 2. Descriptive statistics, factor analysis and internal consistency

Factor analysis gave three factors accounting for 74.2% of the total variation (Table 2). Factor loadings were acceptable with all 15 items having loadings above 0.5 and loadings below 0.3 on the other two factors. In order of the total amount of variation that they explain, the factors can be described as relationship with health personnel, information and participation, and outcome. Table 2 shows that the item-total correlations and Cronbach's alpha for the final scales were acceptable, ranging from 0.65 to 0.89 and from 0.88 to 0.94, respectively.

Table 2 shows the mean scores for the final 15-item questionnaire on a scale of 0 to 100 where 100 is the best possible experiences of care. Scores were skewed towards positive experiences of care and ranged from 59.6 to 74.9 for the scales of information and participation and relationship with health personnel respectively.

The results of validity testing are shown in Table 3. The results of the 45 correlations were all highly significant. The three PEQ-CAMHS scores had moderate levels of correlation with overall satisfaction for how the child and parent were treated, the highest correlations being for the PEQ-CAMHS scale of relationship with health personnel. Slightly higher correlations were found for the child and parent outcomes, the highest being for the PEQ-CAMHS scores of outcome and relationship with health personnel. Moderate to high correlations were found for the two items relating to cooperation and the PEQ-CAMHS scores for relationship to health professional. Low to moderate correlations were found for the two items relating to parent involvement, the highest being for the PEQ-CAMHS scores for relationship with health personnel and information/participation. The item relating to information associated with side effects of medication had the highest correlation with PEQ-CAMHS scores for information and participation at 0.60. As expected the correlations for the variables relating to availability and waiting time had lower correlations, the highest being for PEQ-CAMHS scores of relationship with health personnel. Finally, correlations of a low level were found for responses to the two items relating to whether the health personnel had talked down to them or their child, the highest being for PEQ-CAMHS scores for relationship with health professional.

Table 3. Correlations between questionnaire scores and responses to individual questions

Discussion

This study was part of the first national survey undertaken in Norway to assess parent satisfaction with all Norwegian outpatient CAHMS. The development of the PEQ-CAHMS Outpatients followed a review of the literature including existing questionnaires, interviews with parents, and consultation with an expert group of professionals involved in the delivery of CAMHS and researchers experienced in questionnaire and survey design. This was designed to ensure that the questionnaire covered the important aspects of outpatient care in sufficient detail and hence had content validity. Parents are important participants in the delivery of mental health care to their children [9] and are not only well placed to judge service quality but can also inform the process of development and hence the content of questionnaires that are designed to assess quality of care from their perspective. The resulting questionnaire underwent a rigorous process of piloting and testing for data quality, reliability and validity that have been recommended for evaluating such questionnaires [23].

The 15 items within the PEQ-CAMHS Outpatients contribute to three scales of relationship with health personnel, information and participation, and outcome. The levels of missing data in the range 1.5 to 5.6% suggest that it is acceptable to parents.

The results of factor analysis and tests of internal consistency were strong empirical support for the three scales. The questionnaire has evidence for construct validity following the application of hypotheses based on previous research findings and theory. In this national survey the questionnaire scores were approximately normally distributed in the range of 59.6 to 74.9 which means that there is potential for improvement in parent experiences over time and particularly for the scale of information and participation.

The test-retest reliability of the questionnaire was not assessed due to limited resources. Previous surveys undertaken in Norway have found acceptable levels of test-retest reliability for other questionnaires that assess user experiences [21,27]. Test-retest reliability will be considered for a subsample of parents taking part in a future national survey of user experiences of outpatient CAMHS in Norway. The tests of construct validity were based on other questionnaire items completed by the parents and hence did not include objective or clinical data. Further tests of construct validity should include comparisons with administrative and clinical variables available from the clinics.

The 15 items forming the three scales can be used alone if brevity is an important issue. Having just 15 items makes it more suitable for inclusion in a longer questionnaire alongside other instruments that are designed to assess the outcomes of care for example within a randomised controlled trial and other forms of evaluative study. The additional items included within the questionnaire that related to other aspects of care including information on medication and cooperation between health professionals, should also be considered for inclusion in future surveys. Some of these items were not relevant to all parents but were considered important enough for inclusion in the postal survey by the expert group and hence have content validity.

The questionnaire is being used in national surveys of parent experiences that have included 86 outpatient clinics across Norway and the results for the 2006 survey have been reported [22]. The results are included in the Norwegian national system of quality indicators and are designed to inform parent choice and quality improvement. They are available to parents and clinics in electronic and report form [22].

The questionnaire is designed for use with parents of children attending CAMHS outpatient clinics and while aspects of care that it assesses are of general relevance to CAMHS including the relationship with health personnel, it does not include aspects of inpatient care provision. The development of a questionnaire that included all aspects of CAMHS care would have necessitated a larger study that also included parents whose children had received inpatient care, both as a basis for informing the content of the questionnaire and also in testing. Some important aspects of parent experiences may be relevant for both inpatient and outpatient care and hence a generic questionnaire may be feasible. However, outpatient- and inpatient-specific modules will have greater content validity from a user perspective and are necessary for a more detailed understanding of the quality of CAMHS care. Furthermore, the PEQ-CAMHS Outpatients is designed for parent completion and hence does not include the views of children and adolescents. Future research should also seek to include the views of children and compare these with parent experiences and clinical data including outcomes.

The response rate of 46% lies within the range of 8 to 63% found following a review of surveys of parent satisfaction with child and adolescent mental health; the median response rate to postal surveys was 33% [17]. More recent postal surveys lie in the range 22% to 52% for a survey without reminders [16] and with a telephone reminder [9], respectively. 'Personal contact' or more direct approaches which include surveying parents as part of initial and follow-up procedures at CAMHS produce higher response rates [17] but there is greater potential for social desirability bias [32,33]. Parents may report higher levels of satisfaction because they feel that this will be more acceptable to those administering the survey. Non-response causes bias if non-respondents systematically differ from respondents in relation to important study variables, here parent experiences and satisfaction.

The findings for other follow-up surveys of Norwegian user experiences have shown that the response rates have not caused serious bias [33-36]. The response rate for the telephone survey was 56% following three postal contacts. There was no difference in overall parent experiences for respondents and telephone interviewees as assessed by a summary score comprising four items. However, there was one small but significant difference for one of these four items relating to "viewpoints being taken seriously". There was also a difference in the number of previous contacts with the clinic and education levels of parents. Interviewees reported a greater number of previous contacts and a smaller proportion had received higher education. The former may imply that non-respondents are parents of children with more severe conditions but this requires further investigation.

Parent satisfaction questionnaires have been criticised for lacking evidence relating to quantitative aspects of psychometric testing including reliability and validity [1,9,17]. The evaluation of the PEQ-CAMHS included assessing data quality, factor analysis to assess dimensionality, internal consistency and construct validity that have been recommended in the development of questionnaires designed to measure user experiences and satisfaction [23]. Few studies have undertaken such a comprehensive evaluation which followed consideration of content validity; that the questionnaire includes the important aspects of parent experiences with outpatient CAMHS. Moreover data quality, an important indicator of the acceptability of items, has rarely been reported in previous studies.

Conclusions

The PEQ-CAMHS Outpatients includes important aspects of parent experiences with CAMHS that are based on the views of parents. The questionnaire has evidence for data quality, internal consistency, content validity and construct validity. The PEQ-CAMHS Outpatients is recommended for future applications designed to assess parent experiences of outpatient CAMHS and is being used in national surveys of parent experiences in Norway as a quality indicator. Test-retest reliability and further tests of construct validity, including comparisons with clinical variables are recommended for future research.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

All authors contributed to the design of the questionnaire and survey. AMG conducted the analysis and drafted the manuscript. OAB and OH were involved in data acquisition. All authors have made significant contributions by critically reviewing the manuscript and have read and approved the final version.

Acknowledgements and funding

The authors thank Tomislav Dimoski at the Norwegian Knowledge Centre for Health Services, Oslo, Norway for his contributions including developing software necessary for obtaining data from the CAMHS, conducting the data collection and quality assurance of data. We also thank Saga Høgheim who carried out most of the practical tasks relating to data collection, and members of the expert group that contributed to the development of the questionnaire and survey design. The survey was funded by the Norwegian Department of Health.

References

  1. Riley S, Stromberg A, Clark J: Assessing parental satisfaction with children's mental health services with the youth services survey for families.

    J Child Fam Stud 2005, 14:87-99. Publisher Full Text OpenURL

  2. Garland AF, Haine RA, Boxmeyer CL: Determinates of youth and parent satisfaction in usual care psychotherapy.

    Evaluation and Program Planning 2007, 30:45-54. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  3. Crow R, Gage H, Hampsom S, Hart J, Kimber A, Storey L, Thomas H: The measurement of satisfaction with healthcare: implications for practice from a systematic review of the literature.

    Health Technol Assess 2002., 6(32) OpenURL

  4. Shapiro JP, Welker CJ, Jacobson BJ: The Youth Client Satisfaction Questionnaire: development, construct validation, and factor structure.

    J Clin Child Psychol 1997, 26:87-98. PubMed Abstract | Publisher Full Text OpenURL

  5. Lambert W, Salzer MS, Bickman L: Clinical outcome, consumer satisfaction, and ad hoc ratings of improvement in children's mental health.

    J Consult Clin Psychol 1998, 66:270-9. PubMed Abstract | Publisher Full Text OpenURL

  6. Gerkensmeyer J: Patient satisfaction with mental health services: relationship to outcomes. In The 14th Annual Research Conference Proceedings: A System of Care for Children's Mental Health. Expanding the Research Base. Edited by Liberton C, Kutash K, Friedman R. Tampa, Fla: University of South Florida; 2001:259-266. OpenURL

  7. Kaplan S, Busner J, Chibnall J, Kang G: Consumer Satisfaction at a Child and Adolescent State Psychiatric Hospital.

    Psychiatr Serv 2001, 52:202-6. PubMed Abstract | Publisher Full Text OpenURL

  8. Delaney KR, Engles-Scianna B: Parents' perceptions of their child's emotional illness and psychiatric treatment needs.

    Journal of Child and Adolescent Psychiatric Nursing 1996, 9:15-24. PubMed Abstract OpenURL

  9. Gerkensmeyer JE, Austin JK: Development and testing of a scale measuring parent satisfaction with staff interactions.

    J Behav Health Serv Res 2005, 32:61-73. PubMed Abstract | Publisher Full Text OpenURL

  10. Resendez MG, Quist RM, Matshazi DGM: A longitudinal analysis of family empowerment and client outcomes.

    J Child Fam Stud 2000, 9:449-60. Publisher Full Text OpenURL

  11. Blader JC: Longitudinal Assessment of parental satisfaction with children's psychiatric hospitalization.

    Adm Policy Ment Health 2007, 34:108-15. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  12. Swanson JM, Hechtman L: Using long-acting stimulants: does it change ADHD treatment outcome?

    Can Child Adolesc Psychiatr Rev 2005, 14(Suppl 1):2-3. PubMed Abstract | PubMed Central Full Text OpenURL

  13. Steele M, Weiss M, Swanson J, Wang J, Prinzo RS, Binder CE: A randomized, controlled effectiveness trial of OROS-methylphenidate compared to usual care with immediate-release methylphenidate in attention deficit-hyperactivity disorder.

    Can J Clin Pharmacol 2006, 13:e50-62. PubMed Abstract | Publisher Full Text OpenURL

  14. Martin SJ, Petr GC, Kapp SA: Consumer satisfaction with children's mental health services.

    Child Adolesc Soc Work J 2003, 20:211-26. Publisher Full Text OpenURL

  15. Day C, Davis H: The effectiveness and quality of routine child and adolescent mental health care outreach clinics.

    Br J Clin Psychol 2006, 45:439-52. PubMed Abstract | Publisher Full Text OpenURL

  16. Heflinger CA, Simpkins CG, Scholle SH, Kelleher KJ: Parent/caregiver satisfaction with their child ' s medicaid plan and behavioral health providers.

    Ment Health Serv Res 2004, 6:23-32. PubMed Abstract | Publisher Full Text OpenURL

  17. Young SC, Nicholson J, Davis M: An overview of issues in research on consumer satisfaction with child and adolescent mental health services.

    J Child Fam Stud 1995, 4:219-38. Publisher Full Text OpenURL

  18. Attkisson CC, Greenfield TK: The Client Satisfaction Questionnaire (CSQ) Scale and the Service Satisfaction Scale-30 (SSS-30). In Outcomes assessment in clinical practice. Edited by Sederer LL, Dickey B. Baltimore: Williams & Wilkins; 1996:120-127. OpenURL

  19. Rey JM, Plapp JM, Simpson PL: Parental satisfaction and outcome: a 4-year study in a child and adolescent mental health service.

    Australian and New Zealand Journal of Psychiatry 1999, 33:22-28. PubMed Abstract | Publisher Full Text OpenURL

  20. Ayton AK, Mooney MP, Sillifant K, Powls J, Rasool H: The development of the child and adolescent versions of the Verona Service Satisfaction Scale (CAMHSSS).

    Soc Psychiatr Epidemiol 2007, 42:892-901. Publisher Full Text OpenURL

  21. Garratt AM, Andersen Bjertnæs Ø, Krogstad U, Gulbransen P: The OutPatient Experiences Questionnaire: reliability and validity in 52 Norwegian hospitals.

    Quality and Safety in Health Care 2005, 14:433-37. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  22. Bjertnæs ØA, Garratt A, Helgeland J, Holmboe O, Dahle KA, Hanssen-Bauer K, Røttingen JA: Parent assessment of outpatient child and adolescent mental health services.

    Tidsskr Nor Laegeforen 2008, 128:1041-1045. PubMed Abstract | Publisher Full Text OpenURL

  23. Sitzia J: How valid and reliable are patient satisfaction data? An analysis of 195 studies.

    Int J Qual Health Care 1999, 11:319-328. PubMed Abstract | Publisher Full Text OpenURL

  24. Hair J, Anderson RE, Tatham RL, Black WC: Multivariate data analysis with readings. 4th edition. Englewood Cliffs, New Jersey: Prentice-Hall international Inc; 1995. OpenURL

  25. Nunnally JC, Bernstein IH: Psychometric theory. 3rd edition. New York: McGraw-Hill; 1994. OpenURL

  26. Kline P: The handbook of psychological testing. London: Routledge; 1993. OpenURL

  27. Garratt AM, Andresen Bjertnæs ØA, Barlinn JK: Parent Experiences of Paediatric Care (PEPC) questonnaire: reliability and validity following a national survey.

    Acta Paediatrica 2007, 96:246-252. PubMed Abstract OpenURL

  28. Stallard P: Validity and reliability of the Parent Satisfaction Questionnaire.

    Br J Clin Psychol 1996, 35:311-8. PubMed Abstract OpenURL

  29. Godley S, Fiedler E, Funk R: Consumer satisfaction of parents and their children with child/adolescent mental health services.

    Eval Program Plann 1998, 21:31-45. Publisher Full Text OpenURL

  30. Garland AF, Aarons GA, Saltzman MD, Kruse MI: Correlates of adolescents ' satisfaction with mental health services.

    Ment Health Serv Res 2000, 2:127-39. PubMed Abstract | Publisher Full Text OpenURL

  31. Garratt AM, Bjørngård JH, Dahle KA, Bjertnæs ØA, Saunes IS, Ruud T: Psychiatric Out-Patient Experiences Questionnaire: data quality, reliability and validity in patients attending 90 Norwegian clinics.

    Nordic Journal of Psychiatry 2006, 60:89-96. PubMed Abstract | Publisher Full Text OpenURL

  32. Sitzia J, Wood N: Response rate in patient satisfaction research: an analysis of 210 published studies.

    Int J Qual Health Care 1998, 10:311-317. PubMed Abstract | Publisher Full Text OpenURL

  33. Bjertnæs ØA, Garratt AM, Johannessen JO: Data collection methods and results in users surveys in mental health care.

    Tidsskr Nor Laegeforen 2006, 126:1481-1483. PubMed Abstract | Publisher Full Text OpenURL

  34. Guldvog B, Hofoss D, Pettersen KI, Ebbesen J, Ronning OM: PS-RESKVA -(Patient Satisfaction, Results and Quality -- patient satisfaction in hospitals).

    Tidsskr Nor Laegeforen 1998, 118:386-391. PubMed Abstract OpenURL

  35. Bjertnaes OA, Garratt A, Botten G: Nonresponse bias and cost-effectiveness in a Norwegian survey of family physicians.

    Eval Health Prof 2008, 31:65-80. PubMed Abstract | Publisher Full Text OpenURL

  36. Bjertnaes OA, Garratt A, Iversen H, Ruud T: The association between GP and patient ratings of quality of care at outpatient clinics.

    Fam Pract 2009, 26:384-90. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL