The Journal of Pediatrics
Volume 153, Issue 6 , Pages 733-735, December 2008

Patient Registry Analyses: Seize the Data, but Caveat Lector

  • Michael S. Schechter, MD, MPH

      Affiliations

    • Corresponding Author InformationReprint requests: Michael S. Schechter, MD, MPH, Division of Pulmonary, Allergy/Immunology, Cystic Fibrosis and Sleep, Department of Pediatrics, Emory University School of Medicine, 1547 Clifton Road, Atlanta, GA 30322

Division of Pulmonary, Allergy/Immunology, Cystic Fibrosis and Sleep, Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia

Article Outline

Abbreviations: BMI, Body mass index, CF, Cystic fibrosis, CFF, Cystic Fibrosis Foundation, ESCF, Epidemiologic Study of Cystic Fibrosis, FEV1, Forced expiratory volume in 1 second, ICS, Inhaled corticosteroid

 

Disease-specific registries are observational studies that systematically collect data to follow patterns in disease diagnosis, treatment, and outcomes over time in existing practice settings. When well designed and managed, patient registries are powerful tools for advancing knowledge in the understanding of disease progression and management. Recognizing this, a host of disease specific registries have been developed in this country and elsewhere around the world for tracking such conditions as childhood cancers,1, 2 complications of prematurity,3 inborn errors of metabolism,4 and hemophilia.5

See related articles, p 746 and p 752

Traditionally, disease registries have been established primarily to define the natural history of the disease in question. More recently, additional purposes have evolved, and registries are now being created with the associated purpose of determining clinical effectiveness and safety of new specific therapies6 and measuring quality of care. Interested readers are referred to a handbook for the creation of registries recently produced with the support of the Agency for Healthcare Research and Quality.7

Cystic fibrosis (CF) care has benefited immeasurably from the availability of disease registries to evaluate outcomes. The Cystic Fibrosis Foundation (CFF) sponsored the initial development and adoption of a national patient registry containing demographic and clinical data on patients attending accredited care centers in the United States in the late 1960s. It was initially intended to generate basic descriptive data regarding the CF population, for example, average age of diagnosis, survival, and microbiological information. In the last 15 years, it has been increasingly used for analyses by epidemiologists seeking to identify risk factors and generate hypotheses regarding disease pathogenesis. CFF registry data are available to qualified researchers with an appropriate question and analytical plan through an application process to the CFF, and more than 80 CFF registry analyses have been published to date. The existence of this national patient registry has encouraged individual care centers to maintain their own patient registry databases for research and clinical applications, also resulting in a number of scientific publications.8, 9, 10 More recently, the registry has become an important engine for quality improvement by displaying center-based differences in practice patterns and outcomes. It is now a visit-based, Web-enabled clinical information system that provides care centers with patient- and center-level summary reports on care processes and outcomes and a sophisticated querying tool.11

The Epidemiologic Study of Cystic Fibrosis (ESCF) began in 1993 as an industry-sponsored multicenter, longitudinal, observational study that collected clinical, therapeutic, microbiologic, and lung function data from CF treatment sites in the United States and Canada, and it also served as a phase IV study of dornase alfa.12 Its data collection complemented that of the CFF registry in that ESCF included CF care sites that were not CFF-accredited, included data on medications before the CFF registry did, and later added additional data elements not captured in the CFF registry, such as a validated quality-of-life measure.13 ESCF analyses are coordinated through a Scientific Advisory Group and have focused predominately on (but not been restricted to) examinations of medical practice patterns and how they affect disease outcomes. These analyses have been key to helping determine and promulgate optimal care processes that lead to better disease outcomes for patients with CF.

Disease registries provide data related to risk factors and exposures (including therapeutics) that cannot be readily obtained for either ethical or pragmatic reasons from randomized clinical trials. Regarding therapeutic questions, the real-life data contained in registries allow researchers and clinicians to evaluate whether an intervention found to be efficacious in the setting of a controlled clinical trial becomes effective when placed into general clinical use. The validity of analysis of registry data is threatened by the same pitfalls that are of concern for any observational study, namely selection bias (if patients included in a specific registry analysis are not representative of patients as a whole), information bias (especially if measurements are systematically different for one group compared with another), and confounding (when the apparent relationship between an exposure and an outcome is distorted by the relationship of both to a third factor). Confounding by missing variables is particularly problematic when investigating a research question that was not previously anticipated, because pertinent variables might not have been considered for inclusion at the initiation of the registry. A final hazard in the interpretation of observational data is that causality may be inferred but can never be proven—factors A and B might be associated, but A might cause B, B might cause A, or both might be caused by a third factor. For example, even though patients included in the CF registry who receive inhaled antibiotics have worse forced expiratory volume in 1 second (FEV1) compared with those who do not, it would be a mistake to conclude that inhaled antibiotics make lung function worse. Although some novel approaches may help mitigate the effect of these potential pitfalls,14 threats to validity are typically discussed in work reporting on the analysis of observational studies as a sort of caveat lector (“reader beware”); the reader must do the work of deciding how to interpret the report.

This issue of The Journal contains 2 reports of registry analyses on the effect of treatments on CF outcomes, both of which illustrate the power and limitations of this type of study. Ren et al15 used ESCF data to evaluate the therapeutic effect of inhaled corticosteroids (ICS) on lung function in CF by examining the change in rate of decline in FEV1 before and after initiation of ICS in those patients for whom it was prescribed compared with a control group of patients. They found that the mean decline in FEV1 in patients started on ICS changed from −1.52% predicted/year to −0.44% predicted/year, compared with the control group, which changed from −1.01% predicted/year to −1.44% predicted/year. They also noted that patients treated with ICS showed a drop in height for age percentile and increases in the use of insulin and oral hypoglycemic agents, a finding that might be expected and thus tends to validate their methodology. This research group is experienced in analyzing data from the ESCF registry; they used a statistical model that has been validated by previous usage, adjusted for all potential confounders tracked in the ESCF database, and examined several alternative hypotheses to explain their results. Their findings are important because they contradict those of a recent influential randomized controlled trial,16 and they are compelling, given the power of the analytic methods used. Nonetheless, questions remain regarding their validity. Did selection bias become a factor as the study inclusion criteria led to restricting the analysis to 1208 out of 9803 patients who were started on ICS during the period of interest? Was there confounding due to the greater use of other therapies such as oral antibiotics (an example of an unmeasured variable) in the group started on ICS, because their providers recognized that they were not doing as well? Or is the apparent relationship between FEV1 and ICS use really due to the fact that the control patients had higher FEV1 at the time of the index visit compared with the patients started on ICS and also had experienced a slower deterioration in FEV1 in the preindex period, both predictors of a more rapid drop in FEV1 during the postindex period?17

McPhail et al10 investigated temporal changes in lung function (expressed as FEV1 at age 6 and at age 6 to 12 and rate of decline during that period) and nutrition (expressed as peak body mass index [BMI] percentile, weight for age percentile, and height for age percentile at age 3, 6, and 12 and rates of change during that period) at a single large CF center with its own patient database. Having documented improvements in the later time period (1993-2000) compared with the earlier period (1985-1992), the authors evaluated associated clinical characteristics. They found that better longitudinal peak BMI percentile, less chronic Pseudomonas aeruginosa respiratory tract infection, and the initiation of dornase alfa at least 1 year before lung function measurements was associated with better mean peak FEV1% predicted from age 6 to 12 in the later period, and that better baseline BMI percentile, less BMI percentile downward slope, less chronic P aeruginosa respiratory tract infection, and the initiation of dornase alfa before age 9 were associated with better FEV1% predicted rate of decline in the later period.

The associations are clear, but what do they mean? There might be selection bias, in that patients followed in Cincinnati might be different from those at other centers, but their results are fairly congruent with those from other sources, including the CFF registry. More important are the issues of confounding and causality. The authors tracked the use of dornase alfa but did not have data on other therapies, such as inhaled tobramycin or systemic antibiotics. Thus, the association of better lung function with dornase alfa might be a reflection of more aggressive use of other effective but undocumented interventions that were prescribed for the same patients. Similarly, although better nutrition might lead to better lung function, it is plausible that more aggressive interventions introduced from 1993-2000 (such as closer follow-up with more clinic visits) might lead to improvements in both that are independent of each other. Alternatively, perhaps better lung function leads to better nutrition.

Thus, the dilemma. The associations between variables in these data sets are real and unassailable. The conclusions relating to causality are probably true, but they are not provable using the technique of registry analysis. Registries provide useful descriptive data on outcomes and processes of care that can be used to promote quality of care and for the generation and support of research hypotheses. In the case of rare diseases such as CF, the analysis of patient registries, be they local, national, or international, often is the best, if not the only, source of information on risk factors and the effectiveness of therapies, and as such they are vital sources of data. The development of these registries to bolster clinical knowledge for the treatment of other unusual and chronic conditions of childhood, such as sickle cell disease, spina bifida, and neuromuscular diseases would lead to significant advances in the understanding of these conditions. But these registries must be developed with care, and the potential pitfalls in the analyses of patient registry must be respected, even as their conclusions begin to supply previously unavailable answers to vital clinical questions. Caveat lector.

Back to Article Outline

 

Thanks to Bruce M. Marshall, MD, Cystic Fibrosis Foundation, for his helpful comments on an initial draft of this editorial.

Back to Article Outline

References 

  1. Michel G, von der Weid NX, Zwahlen M, Adam M, Rebholz CE, Kuehni CE. The Swiss Childhood Cancer Registry: rationale, organisation and results for the years 2001-2005. Swiss Med Wkly. 2007;137:502–509
  2. Steele JR, Wellemeyer AS, Hansen MJ, Reaman GH, Ross JA. Childhood Cancer Research Network: a North American Pediatric Cancer Registry. Cancer Epidemiol Biomarkers Prev. 2006;15:1241–1242
  3. Walsh MC, Yao Q, Horbar JD, Carpenter JH, Lee SK, Ohlsson A. Changes in the use of postnatal steroids for bronchopulmonary dysplasia in 3 large neonatal networks. Pediatrics. 2006;118:e1328–e1335
  4. Eng CM, Fletcher J, Wilcox WR, Waldek S, Scott CR, Sillence DO, et al. Fabry disease: baseline medical characteristics of a cohort of 1765 males and females in the Fabry Registry. J Inherit Metab Dis. 2007;30:184–192
  5. Dowling NF, Beckman MG, Manco-Johnson M, Hassell K, Philipp CS, Michaels LA, et al. The US Thrombosis and Hemostasis Centers pilot sites program. J Thromb Thrombolysis. 2007;23:1–7
  6. Pitukcheewanont P, Desrosiers P, Steelman J, Rapaport R, Fuqua JS, Kreher NC, et al. Issues and trends in pediatric growth hormone therapy: an update from the GHMonitor observational registry. Pediatr Endocrinol Rev. 2008;5(Suppl 2):702–707
  7. In:  Gliklich R,  Dreyer N editor. Registries for evaluating patient outcomes: a user's guide. Rockville, MD: Agency for Healthcare Research and Quality; 2007;AHRQ publication 07-EHC001-1 http://effectivehealthcare.ahrq.gov/repFiles/PatOutcomes.pdf
  8. Curtis JR, Burke W, Kassner A, Aitkin ML. Absence of health insurance is associated with decreased life expectancy in patients with cystic fibrosis. Am J Respir Crit Care Med. 1997;155:1921–1924
  9. Schechter MS, Margolis PA. Relationship between socioeconomic status and disease severity in cystic fibrosis. J Pediatrics. 1998;132:260–264
  10. McPhail G, Acton JD, Fenchel MC, Amin RS, Seid M. Improvements in lung function outcomes in children with cystic fibrosis are associated with better nutrition, fewer chronic Pseudomonas aeruginosa infections, and dornase alfa use. J Pediatr. 2008;153:752–757
  11. Schechter MS, Margolis P. Improving subspecialty healthcare: lessons from cystic fibrosis. J Pediatr. 2005;147:295–301
  12. Morgan WJ, Butler SM, Johnson CA, Colin AA, FitzSimmons SC, Geller DE, et al. Epidemiologic study of cystic fibrosis: design and implementation of a prospective, multicenter, observational study of patients with cystic fibrosis in the US and Canada. Pediatr Pulmonol. 1999;28:231–241
  13. Quittner AL, Buu A, Messer MA, Modi AC, Watrous M. Development and validation of The Cystic Fibrosis Questionnaire in the United States: a health-related quality-of-life measure for cystic fibrosis. Chest. 2005;128:2347–2354
  14. Schmoor C, Caputo A, Schumacher M. Evidence from nonrandomized studies: a case study on the estimation of causal effects. Am J Epidemiol. 2008;167:1120–1129
  15. Ren C, Pasta D, Rasouliyan L, Wagener J, Konstan M, Morgan W. Relationship between inhaled corticosteroid therapy and rate of lung function decline in children with cystic fibrosis. J Pediatr. 2008;153:746–751
  16. Balfour-Lynn IM, Lees B, Hall P, Phillips G, Khan M, Flather M, et al. Multicenter randomized controlled trial of withdrawal of inhaled corticosteroids in cystic fibrosis. Am J Respir Crit Care Med. 2006;173:1356–1362
  17. Konstan MW, Morgan WJ, Butler SM, Pasta DJ, Craib ML, Silva SJ, et al. Risk factors for rate of decline in forced expiratory volume in one second in children and adolescents with cystic fibrosis. J Pediatr. 2007;151:134–139

 The author is a member of the North American Scientific Advisory Group of the Epidemiologic Study of Cystic Fibrosis, and receives an annual consulting fee for this activity.

PII: S0022-3476(08)00776-2

doi:10.1016/j.jpeds.2008.09.011

Refers to article:

  • Relationship between Inhaled Corticosteroid Therapy and Rate of Lung Function Decline in Children with Cystic Fibrosis , 01 September 2008

    Clement L. Ren, David J. Pasta, Lawrence Rasouliyan, Jeffrey S. Wagener, Michael W. Konstan, Wayne J. Morgan, Scientific Advisory Group and the Investigators and Coordinators of the Epidemiologic Study of Cystic Fibrosis
    The Journal of Pediatrics December 2008 (Vol. 153, Issue 6, Pages 746-751.e2)

  • Improvements in Lung Function Outcomes in Children with Cystic Fibrosis are Associated with Better Nutrition, Fewer Chronic Pseudomonas aeruginosa Infections, and Dornase Alfa Use , 01 September 2008

    Gary L. McPhail, James D. Acton, Matthew C. Fenchel, Raouf S. Amin, Michael Seid
    The Journal of Pediatrics December 2008 (Vol. 153, Issue 6, Pages 752-757)

The Journal of Pediatrics
Volume 153, Issue 6 , Pages 733-735, December 2008

Article Tools