Confirming the Validity of the General Pediatrics Certification Examinations: A Practice Analysis
Article Outline
ABP, American Board of Pediatrics, GP, General Pediatrics, MOC, Maintenance of Certification
Successful completion of the General Pediatrics (GP) certification examination is one of the requirements for initial certification in GP by the American Board of Pediatrics (ABP). In addition, the continued demonstration of a core set of knowledge, as demonstrated by successful completion of the Maintenance of Certification (MOC) examination, is one of the requirements needed to maintain certification. Test committees comprised of practicing general pediatricians, pediatric subspecialists, and pediatric educators routinely review the content specifications for these examinations. Although this approach is appropriate for validation of the content, national testing standards recommend the demonstration of a relationship between content specifications and practice. The Standards for Accreditation of Certifying Agencies1 specifically state that a practice analysis “must be conducted to clearly delineate performance domains and tasks, associated knowledge and/or skills, and sets of content/item specifications to be used as the basis for developing each type of assessment instruments.” The Standards for Educational and Psychological Tests2 similarly state that “the test specifications should be documented, along with their rationale, and the process by which they were developed.”
The practice analysis process is the most commonly applied and accepted validation strategy used in developing and reassessing credentialing examinations.3 The process not only ensures the validity of the test blueprint by guiding the content distribution in the examination, but it also helps to ensure that the examination is a reliable instrument of measure and that the examination assesses the necessary knowledge and skills to provide high-quality health care for infants, children, and adolescents. This report summarizes the practice analysis study conducted by the ABP for the GP initial and MOC examinations.
Methods
The practice analysis was conducted with subject matter experts who were practicing, board-certified pediatricians. In May 2007, phase 1 of the practice analysis was conducted with a 3-day workshop consisting of 13 subject matter experts who were selected to represent the various geographic areas and practice settings of general pediatricians, along with the expertise needed for both initial certification and MOC examinations. The participants reviewed the current content specifications for the GP examinations and made necessary modifications to ensure that the content was current. They then assessed each of the resulting competency areas by providing ratings regarding importance and frequency of use within the GP profession. In particular, they were asked: (1) How important is this area of knowledge for a board-certified pediatrician?; and (2) How frequent is this area of knowledge used by a board-certified pediatrician? These ratings were then used to calculate the recommended number of questions for each competency area on the examinations.
During phase 2 of the study, a random sample of 1000 general pediatricians was selected from the ABP's database of actively certified diplomates. In August 2007, these general pediatricians received a mailed survey and a detailed content outline of each competency area as finalized in phase 1. The survey also included 7 questions designed to gather information on the respondents and their practice environment, such as length of time in the profession, current work hours, amount of time spent on direct patient care, primary patient population, practice setting, and community size.
The remaining survey consisted of a list of competency areas as determined in phase 1. Participants were directed to rate the competency areas with regard to importance and frequency to their practice. A 5-point Likert scale was used, with 1 indicating very unimportant or not frequently used and 5 indicating critical importance for or high use by a general pediatrician. Analyses of these data will help ensure that the examination blueprint is reflective of actual practice so that competency areas deemed more important and more frequently encountered would have greater coverage on the examination.
A reminder was sent to all in the sample 2 weeks after the initial mailing. Those who did not return the survey received a second mailing of the survey. The data collection was completed in November 2007, ending phase 2.
As the last step, a final review of the competency areas and proposed content specification, as determined by phase 1 and phase 2, was conducted. The current ABP planning committees of the initial certification and MOC examinations served as the final reviewers. The ABP Board of Directors provided the final approval in June 2008.
Results
At the end of phase 1, the workshop participants recommended that the same content distribution or test blueprint be used for both the initial and MOC examinations, because the core knowledge of the profession is the same. However, the types of questions may differ, because one would expect more science-based questions on the initial examination and more clinical-based questions on the MOC examination.
New competency areas were introduced to reflect the changing nature of pediatrics, including patient safety and additional emphasis on ethics, research, cognition/language/learning, and psychosocial issues. The resulting test blueprint included 34 competency areas.
In phase 2, a total of 458 diplomates from 46 states completed the survey. The final response rate was approximately 51%, excluding invalid addresses and disqualified participants, such as those who returned the survey indicating they no longer practice GP. Overall, the respondents reflected a wide range of age, experience, practice type, and geographic settings.
The respondents ranged from 30 to 66 years of age. Approximately 38% of respondents had 10 years or less experience, and 31% had more than 20 years experience. Most (73%) respondents worked full time. On average, respondents spent about 86% of their time on direct patient care, with administration, research, and teaching comprising most of their other time.
About 46% of respondents serve mostly privately insured patient populations, and 24% serve mostly publicly insured or underserved populations. More than 65% of respondents work in small office settings of 2 to 10 individuals. Only 13% of respondents work in solo practice.
Most participants are self-employed or work in privately owned medical groups (63%). Fourteen percent work in an academic heath care center or medical school (14%), with an additional 14% in community or private hospital settings. Most respondents (59%) work in urban regions with larger populations (100 000+), and approximately 20% work in communities with populations less than 50 000.
The participants completed the ratings of the 34 competency areas using a 5-point Likert-type scale. Their ratings were similar to the ratings obtained in phase 1 by the focus group (Table; available at www.jpeds.com).
The results of the 2 phases were presented to the GP Planning Committee and the MOC Planning Committee. At that time, the committees were also asked to consider an additional competency area (Behavioral/Mental Health), which arose on the basis of comments provided during phase 1 and phase 2 of the practice analysis period. Although the content of this new competency area was initially included within the 2 competency areas of Cognition/Language/Attention/Learning and Psychosocial Issues/Problems, it was believed that this area was large enough to merit a separate section within the test blueprint. The committees approved the addition of the new competency area and made the necessary changes to overall distribution of questions for each competency area. The final percent value for each competency area on both the GP initial and MOC examinations is provided in the Table. The ABP Board of Directors approved the final test blueprint and percent distribution in June 2008. This revised test blueprint will be implemented in fall 2009 for both initial certification and MOC, with the actual number of items on the examination matching these defined percentages. The full detailed content specifications can be found on the ABP Web site (www.abp.org).
Discussion
This study included a multiphase approach to validate the competencies necessary for competent performance of a GP. This process allowed input from more than 500 GPs and was conducted within the guidelines of professionally sound testing and measurement practices. Overall, the study resulted in relatively minor changes from the current content specifications, indicating that the current examination adequately reflects practice. However, it did serve to unify the content specifications for initial and MOC, which had been slightly different in the past.
Even though the practice analysis provided a formal, documented link between content and practice, the content specifications are routinely reviewed by ongoing committees as part of the test development process. During these reviews, these committees recommend changes to the specifications on the basis of their perceived requirements of the initial and MOC.
As noted earlier, the purpose of this study was to confirm the content-related validity of the GP initial and MOC examinations by establishing the link between practice and examination content. The results of the practice analysis provide assurance that the examination supports the knowledge necessary for competent patient care of infants, children, and adolescents. It also supports that the ongoing periodic reviews of the content specifications by the currently established test development committees provide an appropriate definition of the GP practice.
Table.
Mean ratings from survey along with final content distribution for general pediatrics initial and maintenance of certification examination (in order by importance mean)
| Domain/Competency Area | Importance mean (n = 456) | Frequency mean (n = 456) | Final content distribution (%) |
|---|---|---|---|
| Growth and developmental milestones | 4.6 | 4.7 | 5 |
| Infectious diseases | 4.6 | 4.5 | 4.5 |
| Preventive pediatrics | 4.4 | 4.5 | 5 |
| Respiratory disorders | 4.4 | 4.4 | 4 |
| Ear, nose, and throat disorders | 4.1 | 4.3 | 4 |
| Nutrition and nutritional disorders | 4.1 | 3.9 | 4 |
| Allergic and immunologic disorders | 4.0 | 4.2 | 3.5 |
| Fetus and newborn infant | 4.0 | 3.9 | 3.5 |
| Disorders of cognition, language, and learning | 4.0 | 3.9 | 3.5 |
| Skin disorders | 3.9 | 4.3 | 3.5 |
| Gastrointestinal disorders | 3.9 | 3.8 | 3.5 |
| Psychosocial issues and problems | 3.8 | 3.8 | 3 |
| Cardiovascular disorders | 3.8 | 2.7 | 3 |
| Emergency care | 3.7 | 2.7 | 3 |
| Adolescent medicine and gynecology | 3.6 | 3.2 | 4 |
| Patient safety and quality improvement | 3.6 | 3.1 | 1.5 |
| Neurologic disorders | 3.6 | 2.9 | 3 |
| Fluid and electrolyte metabolism | 3.6 | 2.8 | 2.5 |
| Endocrine disorders | 3.5 | 2.6 | 3.5 |
| Sports medicine and physical fitness | 3.4 | 3.5 | 2.5 |
| Pharmacology | 3.4 | 3.1 | 2 |
| Disorders of the eye | 3.4 | 2.8 | 1 |
| Poisoning/exposure to hazardous substances | 3.4 | 2.4 | 2 |
| Blood and neoplastic disorders | 3.4 | 2.2 | 2.5 |
| Musculoskeletal disorders | 3.3 | 3.0 | 3 |
| Ethics for primary pediatricians | 3.3 | 2.6 | 1 |
| Substance abuse | 3.2 | 2.5 | 1.5 |
| Renal disorders | 3.2 | 2.2 | 2.5 |
| Critical care | 3.2 | 1.7 | 2 |
| Genetics and dysmorphology | 3.1 | 2.1 | 2.5 |
| Genital system disorders | 3.0 | 2.2 | 1.5 |
| Metabolic disorders | 3.0 | 1.8 | 1.5 |
| Collagen vascular and other multisystem disorders | 2.8 | 1.8 | 1.5 |
| Statistics and research | 2.3 | 1.7 | 1.5 |
| Behavioral and mental health issues | n/a | n/a | 4 |
 Total | 100% |
References
- National Organization for Competency Assurance. National Commission for Certifying Agencies Standards for Accreditation of National Certification Programs, Washington, DC; 2002.
- . American Psychological Association, and National Council on Measurement in Education. Standards for educational and psychological testing. Washington, DC: American Psychological Association; 2007;
- . Job analysis. In: Browning AH, Bugbee AC, Mullins MA editor. Certification: A NOCA Handbook. Washington, DC: National Organization of Competency Assurance; 1996;
PII: S0022-3476(09)00412-0
doi:10.1016/j.jpeds.2009.04.028
© 2009 Mosby, Inc. All rights reserved.
