National Cancer Institute Home at the National Institutes of Health |

Spotlight on Applied Research

Measures & Methods

ARP Collaborating with Multicultural Researchers to Advance Questionnaire Design Methodology

Increasingly, the survey questionnaires used in cancer-related research are administered in contexts that are multi-lingual, multi-cultural, and even multi-national. Adapting to the 'multi-world' demands cross-cultural comparability, and ARP's Dr. Gordon Willis emphasizes that "it is evident that obtaining comparable data across disparate cultural groups is among the most challenging problems facing survey developers." To address this challenge, ARP staff work with a number of researchers, both in the US and internationally, to develop state-of-the-science techniques for (a) survey language translation; and (b) questionnaire pretesting.

Translation: Researchers have typically evaluated survey translations through back-translation, in which an English version is translated to a second language by one translator, the latter is translated back to English by a second, and the two English versions are checked for discrepancies. However, this practice can be problematic. For example, the translation of "I am feeling blue" to Spanish, and then back to English, passes the back-translation test. Yet, the translation is nonsensical, as the Spanish term for blue ("azul") carries no connotation of mood. ARP staff have worked with an international group committed to Comparative Survey Design and Implementation (CSDI)External Web Site Policy to develop an alternative, team-based approach that emphasizes forward (rather than back) translation and the translation of meaning as opposed to wording.

Pretesting: Survey questions also benefit from "product testing" to ensure that they are understood by respondents. A common procedure for such testing is cognitive interviewing, which consists of intensive interviews of members of a target population (Willis, 2005). ARP staff work with a range of researchers (at the U.S. Census Bureau, The University of Michigan, and at Utrecht University in The Netherlands), to adapt cognitive interviewing techniques to the multicultural context. For example, cognitive interviewing demonstrated that the question "In your entire life, have you smoked 100 or more cigarettes?" created an unanticipated difficulty: The phrase "In your entire life" came across as literally "Over your whole life, from birth to death" in Spanish and several Asian languages - a clear miscommunication. The fact that such problems can be remedied (by re-translating to convey "Up until now, have you smoked 100 or more cigarettes?") illustrates the effectiveness of cognitive testing for locating sources of non-comparability, and for pointing toward a means for reconciling them.

ARP plans to continue these collaborative efforts aimed at clear and consistent communication across language, culture, and country. To learn more about ARP's questionnaire design activities, please visit the Web site.

Willis, G. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Sage: Thousand Oaks, CA.

Last Modified: 27 Jun 2014