Key Words: child, adolescent, mental health services, outcome assessment, Psychiatric Department, Hospital, psychometrics/instrumentation
Routine outcome assessment is helpful to inform decision-making, resource allocation, and health policy design. Routine outcome assessment in the hospital setting for children and adolescents with psychiatric disorders remains limited. The clinical instrument HoNOSCA (Health of the Nation Outcome Scales for Children and Adolescents), which has recently become available in Spanish and Catalan, allows outcome assessment in this population from the perspective of patients, their parents or legal guardians, and clinicians. HoNOSCA measures 13 areas of health and psychosocial functioning.
The aim of this study was to assess mental health outcomes in psychiatric day hospital pediatric patients from three perspectives (patient, par-ent/legal guardian, clinician), using the Spanish and Catalan versions of HoNOSCA.
We recruited patients up to 18 years old with any psychiatric disorder at the day unit of the Salut Mental Parc Taulí Hospital Universitari (Sabadell, Catalonia, Spain). We obtained admission and discharge HoNOSCA scores for the patients, their parents or legal guardians, and their clinicians.
We recruited 99 patients over the study period (January 2015 to December 2017), 11 of which were lost to follow-up. Among the remaining 88, we found significant improvement in HoNOSCA scores from admission to discharge. Agreement between the HoNOSCA scores for the three different groups of evaluators (patients, parents/legal guardians, and clinicians) was weak at admission but better at discharge. In general, evaluations from patients and their parents or legal guardians had lower HoNOSCA scores (indicating a better mental health status) at admis-sion compared to those from clinicians. At discharge, however, the scores were more homogenous across the three groups of stakeholders.
Use of HoNOSCA allows for routine evaluation of mental health outcomes in the psychiatric day hospital setting from the perspective of pa-tients, their parents or legal guardians, and clinicians.
Routine outcome assessment in mental health requires instruments that are easily available, brief, and easy to use in the clinical setting. Implementation of these types of instruments allows for reliable and reproducible outcome assessment through quantitative measure of therapeutic change. Routine outcome assessment can help inform decision-making, resource allocation, and policy design,.
One of the most widely accepted instruments for mental health outcome assessment is HoNOS (Health of the Nation Outcome Scales). HoNOS is useful for measuring behavior, impairment, symptoms, and social functioning in mental health patients. A tool specifically designed for children and adolescents, HoNOSCA, has also been developed. HoNOSCA allows researchers to collect assessments from three different stakeholder groups (patients, their parents or legal guardians, and clinicians) through the use of three different scales. The clinician version of the instrument, known as HoNOSCA–Glossary, has two additional items for appraising patients’ and parents/legal guardians’ understanding of the diagnosis and the available management options. HoNOSCA can be administered by various types of professionals, including psychiatrists, nurses, psychologists, counselors, social workers/ educators, occupational therapists, and teachers. Since its publication, HoNOSCA has been translated into several languages, including Danish, French, Lithuanian, and Norwegian,,,,. More recently, the three HoNOSCA scales have become available in Catalan and Spanish. These new versions, translated using a reverse translation approach, were found to have satisfactory internal consistency and inter-rater reliability. The Spanish version of HoNOSCA–Glossary also had good concurrent validity.
Interesting similarities and differences can be found when looking at the perspectives of different groups of informants in health care. In a study exploring treatment preferences in the context of anticoagulation therapy, authors reported wide differences between patients and clinicians on risk of bleeding (threshold risk) for justifying warfarin treatment over aspirin. A similar study found that, compared to clinicians, patients saw more value in avoiding a stroke than in avoiding bleeding. A systematic review of qualitative studies on the use of vitamin K antagonists for atrial fibrillation also showed differences by evaluator group, with patients reporting lack of information and understanding as the main difficulties with anticoagulant treatment, while physicians reported uncertainty about its use, the need for individualized decision-making, and the delegation of responsibility for it (from specialists to family practitioners, for example) as their biggest concern.
Comparisons of multiple perspectives have also been conducted in the context of mental health. A meta-analysis of 341 studies reporting cross-informant correspondence of estimates assessing child and adolescent mental health found low-to-moderate correspondence between stakeholders (parents, children, or teachers); assessments in which the stakeholders being compared observed the child/adolescent in the same setting (e.g., a mother and a father) correlated better than those that did not (e.g., a parent and a teacher). A study on the perspectives of young migrant mental health patients, their parents, and clinicians on optimal health services provision found that patients stressed the importance of ensuring privacy with respect to school or friends, whereas families highlighted addressing measures to decrease stigma. A study of HoNOSCA evaluations by Gowers et al. reported limited agreement between patient and clinician assessments, with Pearson’s correlation coefficients of 0.27 at admission and 0.58 at discharge. Lastly, a recent retrospective pre–post observational study found that patients considered their mental health status to be worse than clinicians did at both admission and discharge.
Despite a growing body of evidence comparing the perspectives of different stakeholders in health care, including mental health, research in this area is still sparse. Assessment of differences in HoNOSCA evaluations from different stakeholders(patients, parents or legal guardians, and clinicians) is also limited, especially for the non-English versions of the instrument. Therefore, this study aimed to assess mental health outcomes in pediatric patients at psychiatric day hospitals from the perspective of patients, parents or legal guardians, and clinicians, using the Spanish and Catalan version of HoNOSCA.
Pediatric patients (18 years of age or less) with at least one of the following diagnoses from the 2017/18 International Classification of Diseases, 10th Revision, Clinical Modification
(ICD-10-CM), section F01–F99 (“Mental, Behavioral and Neurodevelopmental disorders”): psychotic disorders; mood disorders (including major depression disorders, single or recurrent episodes); anxiety disorders; eating disorders; autistic spectrum disorder; and behavioral and emotional disorders. Study inclusion criteria for patients were having parents or legal guardians that were available to participate and the ability of both patients and parents / legal guardians to communicate in Spanish or Catalan.
All patients were recruited at the day unit of Parc Taulí Hospital Universitari (Sabadell, Catalonia, Spain), an outpatient short- to medium-term facility for children and adolescents. This unit has an inter-disciplinary team of social workers/educators, nurses, occupational therapists, counselors, teachers, psychologists, and psychiatrists. In general, patients stay at the unit throughout the whole week, although increasing periods of leave are encouraged throughout the course of admission.
Patients were recruited between January 2015 and December 2017.
We conducted all outcomes assessments using three versions of HoNOSCA: clinician (“HoNOSCA–Glossary”), patient (“HoNOSCA–Patients”), and parent/legal guardian (“HoNOSCA–Parents”).
HoNOSCA measures the following 13 areas of health and psychosocial functioning during the two-week period preceding the assessment: 1) problems with disruptive, antisocial, or aggressive behavior; 2) problems of hyperactivity, attention, or concentration; 3) non-accidental self-harm; 4) problems with alcohol, abuse of substances/solvents; 5) problems with school or language skills; 6) physical illness or disability problems; 7) problems associated with hallucinations, delusions, or abnormal perceptions; 8) problems with non-organic somatic symptoms; 9) problems with emotional and related symptoms; 10) problems with peer relationships; 11) problems with personal care and independence; 12) problems with family life and relationships; and 13) low school attendance. The clinician version of the instrument, HoNOSCA–Glossary, has two additional items: 14) problems in knowing or understanding the nature of the difficulties of the child / adolescent, and 15) problems with lack of information about services or management of child / adolescent difficulties. Each HoNOSCA item is assessed using a 5-point scale ranging from 0 (“no problem”) to 4 (“severe problem”). Higher scores therefore represent greater severity of dysfunction. The final (total) HoNOSCA score is the sum of the scores for all items.
Administration of HoNOSCA and data collection
The HoNOSCA–Glossary instrument was administered by a hospital day unit physician or psychologist once after the first multidisciplinary team case discussion meeting post-admission, where the patient’s background history and recent progress were discussed, and again upon discharge. The HoNOSCA-Parents and HoNOSCA-Patients instruments were also administered twice, to each of those two stakeholder groups (immediately after admission, and upon discharge). Various hospital day unit staff administered these instruments, including psychiatrists, nurses, psychologists, counselors, social workers/ educators, occupational therapists, and teachers.
After the administration of each of the three different HoNOSCA instruments, the interviewers reviewed the medical history of the patient and collected additional information, including demographics and diagnosis, assessed the answers collected, and calculated the final scores.
Training in use of the scales
Participating staff had been trained in the use of the scales during a HoNOSCA validation study. They received all available training material for HoNOSCA, including a workshop video and a training guide. Throughout the course of the study, the lead author (AB) was available to answer any questions about HoNOSCA and address any uncertainty about how to administer the different assessments.
We defined HoNOSCA improvement scores as the difference between measures taken at discharge minus those taken at admission and negative improvement scores as decreases in HoNOSCA scores from admission to discharge (reflecting overall improvement in mental health status). Statistical analyses were performed using repeated-measures ANOVA, with HoNOSCA scores (pre- and post-treatment) and stakeholder group (patient, parent or legal guardian, and clinician) as factors. Subsequently, Bonferroni-corrected pairwise comparisons were conducted. Differences between groups with p values less than 0.05 were considered statistically significant.
The sample size for this study was calculated using Epidat 3.1, accepting an alpha risk of 0.05, a power of 98% in a bilateral contrast, and a correlation coefficient of 0.5. Using this approach, it was found that a total of 80 participants would be needed. Accounting for a lost-to-follow-up rate of 10%, the target sample size was 88.
We obtained approval to conduct this study from the Ethics Committee of the Hospital Clinic of Barcelona, the primary affiliation of the principal investigator of this study (AB). All participants (the pediatric patients and their parents or legal guardians) signed consent forms to participate. All data collected were anonymous and stored in a password-secured database. Data were entered using an anonymous code for each participant in an Excel spreadsheet to facilitate statistical analysis.
A total of 99 children and adolescents were recruited for the study, and 11 of them were lost during follow-up. The characteristics of the 88 patients included in the analysis are shown in Table 1. There were equal numbers of females and males, with a mean age of 14.4 years (standard deviation (SD): 2.00, range 5 to 18) and a median age of 14 years. The most prevalent diagnoses were autistic spectrum disorder, eating disorders, and mood disorders. Mean length of treatment at the day hospital unit was 95.7 days (SD: 73.8, range 11 to 385) with a median of 69 days (Table 1).
Overall, there were significant improvements in the HoNOSCA scores at discharge compared to those obtained at admission (Table 2). The scores varied by diagnosis and stakeholder group. Patients and clinicians conferred higher scores (indicating greater severity of dysfunction) when the diagnosis was autistic spectrum disorder (n = 26) (19.31 and 25.81 respectively) or psychotic disorders (n = 7) (19.71 and 23.71 respectively), with the highest scores from patients corresponding to adjustment disorder (n = 4) (20.5). Parents or legal guardians conferred the highest scores when the diagnoses were behavioral or emotional disorders (n = 13) (23.88 for attention deficit/hyperactivity disorders and 22.40 for other behavioral and emotional disorders). These trends remained constant through discharge, especially for psychotic disorders and behavioral and emotional disorders (Table 2).
Comparing HoNOSCA scores across stakeholder groups, there were differences between clinicians and patients and/or their parents or legal guardians at admission. In general, clinicians conferred higher scores, meaning that they perceived patients' health status to be worse than reported by the patients themselves and/or their parents or legal guardians (Tables 2 and 3). Differences between groups were considered statistically significant when the p values were less than 0.05.
At discharge, scores decreased and became more homogeneous, becoming almost equal across all three stakeholder groups (Table 3 and Figure 1).
Measuring relevant outcome variables in mental health facilitates assessment of the efficacy, strengths, and limitations of the services provided In this study we measured health outcomes in day hospital patients from the perspective of patients, their parents or legal guardians, and clinicians, using HoNOSCA. Our work shows that HoNOSCA is a viable tool to achieve this goal in a systematic and reproducible way in the day hospital psychiatric hospital setting. The improvement in scores from admission to discharge is consistent with the expected improvement after therapeutic treatment. However, given the ample variability of diagnosis and the sample size calculation we implemented, this study did not have sufficient power to detect differences in HoNOSCA scores by diagnosis. We therefore report the results we obtained but make no inferences regarding this data.
At admission, there were significant differences in the assessment of mental health status by clinicians compared to patients and their parents or legal guardians. At discharge, however, the appraisals became more homogeneous (Table 3). This phenomenon speaks positively about the treatment and psychoeducational work provided by the health professionals was effective. We cannot assess whether the differences between the admission and discharge scores were relevant to patients, however, since no studies have assessed the minimal important difference (MID) for HoNOSCA. MIDs are defined as “the smallest difference that patients perceive as beneficial and that would mandate, in the absence of troublesome side effects and excessive cost, a change in the patient’s management” and facilitate the interpretation of patient-reported outcomes.
The reasons behind the differences between assessments from patients and parents or legal guardians and clinicians are unclear. On the one hand, patients may be able to better understand their condition, especially in the less acute, outpatient setting studied here, especially at admission, before clinicians have fully grasped patients’ overall condition. Over the span of the hospital stay, however, clinicians may become better informed of the patient’s complete medical history and condition,. On the other hand, clinicians’ medical training may equip them well to provide a more accurate assessment of a patient’s status.
Our results are in line with those from previous studies that report improvement in HoNOSCA scores after therapeutic interactions and at discharge,,. Like our study, Gowers et al. found that HoNOSCA scores from clinicians were generally higher than those from patients in the first assessment and in the early stages of the interaction, but over time and at discharge the scores became more homogenized. This finding was true both for inpatient and outpatient settings. Different findings emerged in a more recent study at a National Health System tier 4 adolescent inpatient unit for youth in the UK, however, where patients considered their mental health status to be worse than clinicians did, according to the HoNOSCA instrument, at both intake and discharge.
Strengths and limitations
The main strength of our study is the large sample size we were able to recruit. Another strength was the extensive experience the hospital personnel responsible for administering HoNOSCA had with the scale administration due to their participation in a previous translation and validation study of this tool11. The main limitation of our study was that it was restricted to a single health facility in Catalonia, thus potentially compromising its external validity. In addition, this study was limited to a day hospital setting; results could be different in intensive care and other clinical settings.
Implications for practice
The Spanish and Catalan versions of HoNOSCA have proven to be useful tools for outcome measurement in children and adolescents in the day hospital setting. Use of HoNOSCA is feasible within this setting, which allows for assessment from the perspective of patients, parents or legal guardians, and clinicians. HoNOSCA can also be useful for gauging level of severity and improvement in patients with different diagnoses and for promoting evidence-based decision-making in day hospital settings.
Implications for research
The results we present here highlight differences in outcome assessment between clinicians and patients and legal guardians in a day hospital setting. It would be interesting to explore whether these differences persist in other contexts, such as full-hospitalization settings. In addition, our work is limited to a single health facility; additional research to confirm these findings across multiple facilities would be useful. Further research could also explore, in depth, the reasons for the differences in the assessments from the three different stakeholder groups reported here.
This work was not designed to draw any conclusions regarding sensitivity to change in the Spanish and Catalan versions of HoNOSCA. This is the object of another research project currently under way. Lastly, the MID for HoNOSCA could be explored using an anchor-based approach that incorporates an independent standard that both patients and clinicians can recognize and that can provide a measure of small but important differences correlated to HoNOSCA.
HoNOSCA allows for mental health outcome assessment in the day hospital setting not only from the perspective of clinicians but also from the perspective of patients’ parents or legal guardians and, even more important, the patients themselves.
AB: Study design, data collection, statistical analysis, first draft of the manuscript, approval of the final version of the manuscript. JT: Study design, data collection, approval of the final version of the manuscript. PM: Study design, statistical analysis, approval of the final version of the manuscript. HP: Study design, statistical analysis, first draft of the manuscript, approval of the final version of the manuscript.
Anna Ballesteros-Urpí is a PhD candidate in the Education Department at the University of Barcelona (UB) in Barcelona, Spain.
We want to acknowledge Eloísa Arias Núñez, Montserrat Pàmias Massana, and Joaquim Puntí Vidal from Parc Taulí Hospital Universitari for their help during data collection. We also want to thank Andreu Ferrero-Gregori and Francesc Pla Juncà for their help with the statistical analysis.
The authors declare that there are no conflicts of interest involving this work.
This study was self-funded.
Approval to conduct this study was obtained from the Hospital Clinic de Barcelona Ethics Committee (# HCB/2014/0337).
All data from this study are available upon request from the corresponding author.
From the editors
The original version of this manuscript was submitted in Spanish. The English translation was copyedited by the journal.
La evaluación rutinaria de variables de resultado ayuda en la toma de decisiones, la asignación de recursos y el diseño de políticas en salud. La evaluación rutinaria de variables de resultado en el entorno hospitalario para niños y adolescentes con trastornos psiquiátricos sigue siendo limitada. La Health of the Nation Outcome Scales for Children and Adolescents, HoNOSCA, que recientemente se ha traducido al español y al catalán, permite la evaluación de resultados en esta población desde la perspectiva de pacientes, padres o tutores legales y clínicos. Este instrumento mide 13 áreas de salud y funcionamiento psicosocial.
Evaluar variables de resultado en salud mental entre pacientes jóvenes de hospital de día de psiquiatría desde estas tres perspectivas usando Health of the Nation Outcome Scales for Children and Adolescents.
Reclutamos pacientes pediátricos (18 años o menos) con cualquier trastorno psiquiátrico en el hospital de día de la unidad de psiquiatría y psicología juvenil del Hospital Salut Mental Parc Taulí (Sabadell, Cataluña, España). Obtuvimos puntuaciones de Health of the Nation Outcome Scales for Children and Adolescents desde la perspectiva de pacientes, sus padres o tutores legales y clínicos, al ingreso y al alta.
Reclutamos 99 pacientes entre enero de 2015 y diciembre de 2017; once se perdieron durante el seguimiento. Entre los 88 restantes, encontramos una mejora significativa en las puntuaciones de Health of the Nation Outcome Scales for Children and Adolescents desde el ingreso hasta el alta. El acuerdo en las puntuaciones de esta escala entre los diferentes evaluadores fue débil al inicio, pero mejor al alta. En general, los pacientes y los padres o tutores legales reportaron puntuaciones más bajas de Health of the Nation Outcome Scales for Children and Adolescents (mejor estado de salud) al ingreso en comparación con los profesionales de la salud. Al alta, las puntuaciones fueron homogéneas desde las tres perspectivas.
La Health of the Nation Outcome Scales for Children and Adolescents permite la evaluación rutinaria de variables de resultado en salud mental en el entorno de hospital de día de psiquiatría desde la perspectiva de los pacientes, sus padres o tutores legales y los profesionales de la salud.
Citation: Ballesteros-Urpí A, Torralbas-Ortega J, Muro P, Pardo-Hernandez H. Measure of clinical improvement in children and adolescents with psychiatric disorders: an evaluation of multiple perspectives with HoNOSCA . Medwave 2020;20(1):e7762 doi: 10.5867/medwave.2020.01.7762
Submission date: 14/10/2019
Acceptance date: 14/12/2019
Publication date: 13/1/2020
Origin: Not commissioned
Type of review: Externally peer-reviewed by one reviewer, double blind
We are pleased to have your comment on one of our articles. Your comment will be published as soon as it is posted. However, Medwave reserves the right to remove it later if the editors consider your comment to be: offensive in some sense, irrelevant, trivial, contains grammatical mistakes, contains political harangues, appears to be advertising, contains data from a particular person or suggests the need for changes in practice in terms of diagnostic, preventive or therapeutic interventions, if that evidence has not previously been published in a peer-reviewed journal.
No comments on this article.
To comment please log in