National Cancer Institute Home at the National Institutes of Health | www.cancer.gov

Conference Program, Summary Papers, & Presentations

Summary papers and presentations from conference experts are provided below. Access a table of key reference terms (PDF, 123 KB) for the papers.

Pre-Conference Introductory Workshop

Wednesday, June 23, 1:00-5:00 pm

Item Response Theory (IRT) and Applications of IRT for Improving Health Outcomes Measurement

Instructor:
Steve P. Reise, PhD
Professor
UCLA Department of Psychology

This introductory workshop began with a discussion of the basics of item response theory (IRT) modeling, including discussions of defining model parameters, model assumptions, item and test characteristic and information curves, frequently-used IRT models, and differences between traditional and IRT methods. The course discussed the role of IRT modeling in health outcomes research of evaluating and developing questionnaires, identifying differential item functioning, linking instruments, developing item banks, and assessing change. Critical issues for applying this methodology in health outcomes research was discussed.

Start of Scientific Meeting

Day One (Thursday, June 24)

8:00 am

Welcome and Opening Remarks

Ron D. Hays, PhD
Professor, David Geffen School of Medicine at UCLA
Senior Scientist, RAND Corporation

8:20 am

Conference Charge

Joseph Lipscomb PhD
Branch Chief, Outcomes Research
National Cancer Institute

Eleanor M. Perfetto, PhD
President, Drug Information Association
The Weinberg Group, Inc.

8:40 am

The Science of Health Outcomes Measurement

Neil K. Aaronson, PhD
Head, Division of Psychosocial Research & Epidemiology
The Netherlands Cancer Institute

This session reviewed the current state of the science in health outcomes measurement. What defines a "quality" or psychometrically-strong measure? What are the limitations of the instruments we use today? Are we satisfied with the analytical tools used to evaluate our measures?

9:20 am

The Traditional and Modern Approaches to Outcomes Measurement

Ronald K. Hambleton, PhD
School of Education, University of Massachusetts at Amherst

This session defined and contrasted both the traditional approach (i.e., CTT) and the modern measurement (i.e., IRT) approach for data analysis and scoring. Also, it discussed the limitations and strengths of these methods in health outcomes measurement.

10:10 am

Refreshment Break

10:40 am

Building and Revising Outcome Measures: Evaluating Item and Scale Functioning with the IRT model

Bryce B. Reeve, PhD
Psychometrician, Outcomes Research
National Cancer Institute

Maria Orlando, PhD
Behavioral Scientist
RAND Corporation

This session discussed how IRT may be used to evaluate both the properties of the items and the scale and to inform questionnaire revisions. What added information do we learn about the data from IRT modeling?

11:40 am

Assessing Measurement Equivalence Across Populations: Differential Item Functioning (DIF)

Jeanne A. Teresi, EdD, PhD
Senior Research Associate
Columbia University

Leo S. Morales, MD, PhD
Assistant Professor, David Geffen School of Medicine at UCLA
Behavioral Scientist, RAND Corporation

John A. Fleishman, PhD
Senior Social Scientist, Center for Financing Access and Cost Trends
Agency for Healthcare Research and Quality (AHRQ)

This session discussed the importance for evaluating DIF for determining measurement equivalence of item content across instruments administered to different populations (differing by race, age, gender, etc.) or instruments administered in different languages. How will DIF assessment benefit health disparities research?

1:00 PM

Luncheon and Review Posters

2:20 PM

Comparing or Combining Scores from Multiple Instruments: Instrument Linking (Equating)

Neil J. Dorans, PhD
Principal Measurement Statistician in the Research & Development Division
Educational Testing Service

This presentation addressed several issues associated with the linking of health outcomes. What is meant by outcomes linking and equating? How does equating differ from other types of linking? What are common data collection designs used to capture data for outcomes linking. What are some of the standard statistical procedures used to link outcomes directly? What assumptions do they make? What role does IRT play in linking outcomes? What assumptions do IRT methods make? Illustrations of different kinds of linkages were provided.

3:00 PM

Developing Tailored Instruments: Item Banking and Computerized Adaptive Assessment

Chih-Hung Chang, PhD
Assistant Professor
Northwestern University

Jakob B. Bjorner, MD, PhD
Deputy Chief Science Officer
QualityMetric Incorporated

David Thissen, PhD
Professor, Quantitative Program, Department of Psychology
University of North Carolina at Chapel Hill

This session discussed the critical role that IRT plays to develop the item bank and why CTT methods are inappropriate for such a task. What challenges/decisions will we have to make in choosing both the appropriate IRT model and the constructs we want to measure? How can we balance the assumptions of the unidimensional IRT models with the multifaceted nature of the constructs measured in health outcomes research?

4:45 PM

Reception and Poster Session

Day Two (Friday, June 25)

8:00 am

The IRT Data Analysis Project: In parallel sessions, two research teams demonstrated the strengths and challenges of applying IRT modeling for evaluating data in health outcomes research and behavioral science. The sessions provided detailed examples of the logic and flow of analyses, the tackling of technical problems, and the interpretation of results. Examples of input and output files along with analysis summaries were provided to the audience.

Demonstrate item and scale analysis, differential item functioning, instrument linking, and item banking using data collected on four HRQOL questionnaires. Three applied presentations using Behavioral Consortium data to illustrate IRT applications.
Research Team:
Karon F. Cook, PhD, VA (Team Leader)
Cayla R. Teal, PhD, VA (Team Leader)
Jakob B. Bjorner, MD, PhD, QualityMetric
David Cella, PhD, Northwestern U.
Paul K. Crane, MD, MPH, U. of Washington
Chih-Hung Chang, PhD, Northwestern U.
Laura Gibbons, PhD, U. of Washington
Ron D. Hays, PhD, UCLA, RAND
Colleen A. McHorney, PhD, Indiana U.
Katja Ocepek-Welikson, M.Phil., NYC Hebrew Home for the Aged
Anastasia E. Raczek, PhD, QualityMetric
Bryce B. Reeve, PhD, NCI
Jeanne A. Teresi, EdD, PhD, Columbia U.
Research Team:
Louise C. Mâsse, PhD, NCI
Mark R. Wilson, PhD, U.C. Berkeley
Diane D. Allen, U.C. Berkeley

10:00 am

Refreshment Break

10:30 am

The Future of Outcomes Measurement: Item Banking, Tailored Short-Forms, and Computerized-Adaptive Assessment

David Cella, PhD
Professor of Psychiatry and Behavioral Science, Northwestern University Medical School
Director, Center on Outcomes Research and Education

John E. Ware, PhD
Chief Executive Officer and Chief Science Officer
QualityMetric Incorporated

This session discussed how computerized-adaptive testing (CAT) will revolutionize the way we measure outcomes through the development of tailored short forms and computerized-adaptive tests. The speakers presented results from their own work and discussed how item banking (short forms and CATs) will benefit the measurement of patient-reported outcomes in observational studies, clinical trials, and clinical practice.

12:00 PM

Luncheon

1:15 PM

Critical Issues for Developing and Maintaining Item Banks and CATs
Bringing the Concept into Reality: The Idea of a National Item Bank

Colleen A. McHorney, PhD
Professor, Indiana University School of Medicine
Senior Scientist, Regenstrief Institute, Inc.

Jeff A. Sloan, PhD
Lead Statistician and Chair of Quality of Life Research Committee
Mayo Clinic, Rochester

Dennis A. Revicki, PhD
Director - Center for Health Outcomes Research & Sr. Research Leader, MEDTAP Int.
Adjunct Professor, Department of Psychiatry, Georgetown University Medical School.

Lawrence J. Fine, MD, DrPH
Medical Advisor, Office of Behavioral and Social Sciences Research
National Institutes of Health

Robert T. O'Neill, PhD
Director, Office of Biometrics,
Center for Drug Evaluation and Research, FDA

Laurie B. Burke, RPh, MPH
Director, Study Endpoints and Label Development
Office of New Drugs, Center for Drug Evaluation and Research, FDA

This session discussed the feasibility for developing and implementing item banks and CATs in observational and clinical settings. What critical issues should we address and how can we overcome the barriers to issues like proprietary rights? What are the costs and amount of work that are required to create these products? What role should government (e.g., NIH, FDA) and private entities play in this effort?

3:15 PM

Refreshment Break

3:30 PM

Next Steps in Health Outcomes and Behavioral Science

Ron D. Hays, PhD
Professor, David Geffen School of Medicine at UCLA
Senior Scientist, RAND Corporation

Margaret Rothman, PhD
Executive Director, Health Economics
Johnson & Johnson

Joseph Lipscomb PhD
Branch Chief, Outcomes Research
National Cancer Institute

Peter M. Fayers, PhD
Professor of Medical Statistics
University of Aberdeen Medical School

For both health outcomes and behavioral researchers, speakers made recommendations for how their field of study will benefit by incorporating the methods discussed over the past two days. What demonstrations are necessary to explore the potential benefits of item banking and CATs to health outcomes measurement? What role should academia, industry, and government play to push this arena forward?

Last Modified: 03 Sep 2013