Departing from the assessment protocol 14, literature searches for assessment did not include "abstracts, ... conference proceedings, ... and other grey literature etc." (see final protocol, pp. 2f. 14; cf. also TAR, p. 178), thus excluding relevant cost-effectiveness analyses in the public domain at the time of assessment 17181920. This notwithstanding, it was claimed that "this review presents a comprehensive overview of existing economic evaluations of methylphenidate, atomoxetine and dexamphetamine for children and adolescents with ADHD" (TAR, p. 266). Further search anomalies include the overlooking of at least two clinical studies meeting the specified inclusion criteria 2122.

For assessment, studies had to have a minimum duration of three weeks because "the literature suggests that three weeks is the minimum duration for therapeutic trials" to assess "the impact on the social adjustment of the child" (TAR, p. 44), citing the DSM-IV diagnostic manual (TAR, p. 45). The rationale offered was that "the effect of medication on behaviour is often (not always) apparent immediately, but the impact on the social adjustment of the child my well not be apparent in the first days of therapy (final assessment protocol 14, pp. 3ff., and TAR, p. 45). To make sense out of this reasoning, one would expect a minimum treatment duration of three weeks. However, a minimum study duration of three weeks was applied as inclusion criterion. As a consequence, more than one third of the 64 trials included in the clinical effectiveness review were very-short-term crossover studies with treatment periods of one week or less, some of which had been conducted without washout phases between treatment periods (TAR, pp. 51–163) 11.

The observation that the choice of outcomes measures reflects a critical design choice for analysis (TAR, p. 178) was not followed by a review of the literature on measurement instruments 2324. Although social adjustment of patients was implied to be the outcome of interest (TAR, p. 45), clinical effect measures reflecting functional impairment were discarded from analysis (TAR, p. 46). Instead, clinical global impressions (CGI scores) were used "as a proxy of quality of life" (TAR, pp. 16 and 48), and CGI-I (improvement) subscores were selected as the primary endpoint informing economic evaluation. In this respect, the economic analysis deviated from the clinical effectiveness review that had included measures of hyperactivity and comprised a total of 64 randomized clinical trials (RCTs) plus the MTA study 2526. This decision was motivated by the desire to compute quality-adjusted life years (QALYs) as effectiveness measure for economic modeling (TAR, pp. 224f.). Since this maneuver left only five RCTs with treatment durations from three to eight weeks for modeling, and none of those included dexamphetamine, a study previously excluded for inadequate data presentation was secondarily added in order to have any data on dexamphetamine for economic analysis (TAR, pp. 225f. and p. 338). Given well-documented gender differences in ADHD 27, which include clinical response to methylphenidate 28, and the fact that the disorder is most often diagnosed in boys 29, it is noteworthy that this three-weeks cross-over study had reported on 32 girls 30. The assessment did not offer a discussion of this peculiarity. On this basis, after eliminating consideration of the role of concomitant psychosocial interventions, 19 alternative treatment "strategies" (in fact, product sequences) were modeled (TAR, p. 221).

Another important gap occurred in relation to atomoxetine. First, one out of two state-of-the-art RCTs comparing atomoxetine and long-acting methylphenidate was overlooked 2231. These studies concurred suggesting lower or at best equal efficacy of atomoxetine 22313233. Second, two analyses were not considered that provided effect size estimates for long-acting stimulants (0.95 – 1.02) and nonstimulant medications (0.44 – 0.62), using core symptom improvement as effect measure 34353637. These findings had been interpreted by their authors as "substantial and significant differences in efficacy" 37.

In an attempt to overcome the limitations of the remaining database, the NICE assessment relied on advanced mixed-treatment comparison techniques for quantitative meta-analysis of response rates, which were subsequently transformed into QALY gains. For QALY computation, quality weights were derived from utility studies that had used health state descriptions, which did neither correspond to the CGI criteria nor to any of the other clinical endpoints secondarily added (cf. TAR, pp. 359ff.). This approach was pursued despite explicit recognition that "the validity of these measures depends on the content and style of the vignette used to describe each health state" (TAR, p. 181).

In order to broaden the database of six RCTs, data derived from different clinical effect measures were subsequently pooled for "sensitivity analyses"; data synthesis comprised heterogeneously defined "response rates" based on (in addition to the CGI-I subscores) CGI-S ratings as well SNAP-IV and ADHD-RS scores (TAR, p. 254), while the most widely used measures of clinical efficacy in ADHD trials, the Conners Rating Scales 2324, remained excluded from economic modeling (TAR, p. 224) despite their well-documented psychometric properties 24. These narrow-band symptom scales (i.e., the SNAP-IV and ADHD-RS 2324) were erroneously regarded as "disease-specific instruments [measuring] health-related quality of life in children" (TAR, p. 176). In total, these secondary extensions resulted in the inclusion of 13 RCTs, four of which were designated "commercial-in-confidence" and not disclosed (TAR, Chapter 6).

In a final step, also the MTA study 2526 – arguably the most important clinical study completed in ADHD to date – was added, although it remains enigmatic which data were actually used, as the model used information from three out of the four study arms only (TAR, p. 254): Whereas it was stated that "the nature of the treatment received in the community comparison arm of the MTA trial is still unclear, and as a result this data is omitted from the analysis" (TAR, p. 254), a table on the same page of the assessment report explicates that "results for behavioral treatment were omitted as not relevant to this review" (TAR, p. 254).

Whereas the assessment team did not explain its implicit assumption that CGI-I subscores – its primary measure of effectiveness used for the calculation of "response rates", which directly refers to a comparison "to the patient's condition before admission to the project" 38 – were independent from baseline, it rejected the Conners Rating Scales – the most widely used group of measurement instruments in ADHD studies 24 – for precisely this reason (TAR, p. 186 and p. 224). As a consequence, (apart from the enigmatic use of MTA study data) none of the 14 extended treatment studies reviewed by Schachar et al. (2002) 39 were included in data synthesis for cost-effectiveness evaluation (cf. TAR, Chapter 6). Also insights from an important 24-months RCT involving more than 100 patients treated with methylphenidate 4041 were not considered because the study did not fit the narrowly defined inclusion criteria for review. Finally, discussion in the assessment report of the 24-months follow-up data from the MTA study, providing insights into the persistence of treatment effects over the first ten months after trial completion 42, was limited to the clinical review (TAR, p. 168); these data were not addressed in the context of the economic modeling exercise (TAR, Chapter 6).

Although the assessment group correctly observed that MTA subgroup analyses "should be seen as 'exploratory', because of the danger of repeated statistical testing with a sample not designed for this purpose" (TAR, p. 167), it claimed at the same time that its own "model is probabilistic, meaning that relevant input parameters are entered as probabilistic distributions in order to represent the uncertainty around each point estimate" (TAR, p. 220), emphasizing that "the output from the model incorporates the uncertainty around the estimated response rates" (TAR, p. 229). However, in RCTs it is the primary analysis, as defined ex ante, which is most important, and the CGI scores did not represent the primary endpoint in any of the studies selected for synthesis. While it is certainly legitimate to carry out secondary analyses, these should not be represented as fully capturing stochastic uncertainty 43.

So-derived differences in QALYs gained by each treatment "strategy" extended to the third or fourth decimal place only (TAR, pp. 237ff.), and the primary analysis produced a series of inconsistent rankings (TAR, p. 237), which were left uncommented and disappeared only after secondary model extensions ("sensitivity analyses", TAR, pp. 240ff.) comprising the pooling of heterogeneous response criteria mentioned earlier. Although it was claimed that "the issue of heterogeneity was overcome by basing the base case [primary] analysis on trials that are more similar in terms of how they measure the outcome of interest" (TAR, p. 266), in fact internally consistent model results were achieved after this pooling only, and there is no indication that potential confounding effects between treatment strategies and effect measures were assessed.

An important issue pervading the NICE assessment is the way the distinction between efficacy and effectiveness was (not) addressed. Whereas RCTs follow an explanatory orientation ("Can the intervention work?"), economic evaluations to be meaningful require a pragmatic orientation ("Does the intervention work?") 164445. Efficacy data collected during RCTs deliberately and necessarily exclude naturalistic effects associated with a routine clinical practice setting, while effectiveness may be influenced by a number of external factors, notably including poor treatment compliance. "Artificially enhanced compliance" in RCTs has come to be recognized as a major threat to their external validity 46. This is a relevant aspect since the more expensive treatment options evaluated (atomoxetine and modified-release formulations of methylphenidate) differ from their comparators by their simplified dosage regimens, which may be expected to result in improved treatment adherence in practice settings.

There are multiple streams of evidence supporting this expectation. First, there is a statistically significant association between complexity of dosage regimens and treatment compliance 47. Second, there are reasons to assume that treatment adherence of patients with ADHD may be impaired by disease-specific factors 4849. Third, due to their pharmacokinetic and pharmacodynamic properties, the behavioral effects of short-acting stimulants dissipate rapidly after three to four hours – making these drugs prototypical examples of non-forgiving compounds regarding noncompliance 505152535455. Fourth, three independent retrospective database studies consistently indicate higher treatment persistence rates for patients receiving long-acting stimulants compared to short-acting formulations 5657585960. Although analyses based on administrative data typically do not allow differential analysis of reasons for treatment discontinuation and may be distorted by patient selection bias, data showing a higher number of prior diagnoses and significantly lower rates of accidents, injuries, emergency room visits, and hospitalizations among those treated with long-acting formulations are consistent with the assumption that such distortions were absent 565960. Fifth, mediator analyses of the NIMH MTA study confirm the important role of compliance: as intended acceptance/attendance was found to significantly enhance treatment response for both the medication management and combined treatment strategies 26. The NICE assessment group overlooked these findings (TAR, pp. 167ff.) and reasoned instead that "we can also incorporate the results of the MTA trial, but only by assuming that the medical management group in that trial represents treatment with immediate-release methylphenidate" (TAR, p. 253).

For economic evaluation, there are two broadly accepted ways to address the impact of compliance 6162, i.e., the use of decision analytic models combined with appropriate sensitivity analyses and information from randomized "pragmatic trials" with minimal study management 63. Modeling studies, sixth, have been indicative of an acceptable cost-effectiveness of long-acting methylphenidate, possibly reaching extending dominance over short-acting formulations 171864. Seventh, a randomized open-label study comparing long-acting with short-acting methylphenidate reported a number-needed-to-treat (NNT) of 3.6 to 4.8 to achieve one additional responder (depending on response criterion applied), consistently below the NNTs synthesized for assessment (TAR, pp. 226ff.) 116566, although this particular study was impaired by the absence of teacher-reported outcome ratings.

None of these aspects are reflected in the NICE assessment. Instead it was "assumed that the trial data [referring to double-blind, double-dummy ADHD trials] adequately captures the effect of compliance on response to treatment" (TAR, p. 232). As a consequence, data from highly controlled double-blind RCTs and from randomized pragmatic open-label studies were pooled, necessarily concealing any differentiation on grounds of treatment compliance. Nevertheless it was claimed "the effect of compliance on response rates [...] is reflected in the model" (TAR, p. 250).

The economic model (cf. Fig. 1) relied on cost per QALY estimates based on response rates, using utility weights taken from one study reporting parent-proxy ratings using the EQ-5D 67 but not from standard gamble experiments as stated in the assessment report (TAR, p. 235). Although the structure of the model implied assumptions on withdrawal rates, which caused double-counting of nonresponders (TAR, p. 230), no attention was paid to the uneven effect this modeling approach had on the treatment options evaluated: the fact that for dexamphetamine extremely low withdrawal rates were assumed (TAR, p. 236, solely based on the study by Sharp et al., 1999 30, with n = 32 girls observed over three weeks, which had initially been excluded: TAR, p. 231; cf. also TAR, pp. 225f. and p. 338) could only bias the modeled "treatment continuation rates" (cf. TAR, p. 222f.) in favor of dexamphetamine. This source of bias remained unmentioned.

None of the studies selected for the primary model exceeded an observation period of eight weeks per treatment arm (TAR, p. 226), and also the secondary model extensions (referred to as "sensitivity analyses") were informed by trials with observations periods of 12 weeks or less, except for the data extracted from three out of four parallel arms of the MTA study (TAR, p. 254). On this basis, costs and benefits were extrapolated over a time horizon of 12 years. Although the assessment includes a discussion of sensitivity of findings to time horizon (TAR, pp. 245ff.), this did not include a review of long-term sequelae associated with ADHD. No attempt was made to address the impact of the disorder on educational outcomes, injuries and accidents, or other problems such as encounters with the criminal justice system and the burden of caregivers.

The technology assessment purports to have "clearly identified an optimal treatment strategy" (TAR, pp. 19, 261, 266). The caveats offered essentially relate to a paucity of evidence and poor reporting of studies (e.g., TAR, pp. 18ff., 266), which were blamed for the inability to discriminate between drugs in efficacy or between patients in terms of ADHD subtype, age, gender or previous treatment (TAR, pp. 266f.).