Response rate trends in Japan: Examination of three governmental
public opinion surveys
Nicolaos E. Synodinos (University of Hawaii)
&
Shigeru Yamada (posthumous, Kokushikan University)
Over the years, several authors (Ando, 2012; Ando, 2009; Aramaki, 2010; Hino, 2015; Konuki, 1985; Onodera, 1995; Shinoki, 2010; Sugiyama, 1992; Synodinos & Yamada, 2000; Tsuruta, 2008; Ujiie & Takeshita, 1995; Yoshikawa, 2010) discussed the decline in the response rates of Japanese public opinion surveys. These studies investigated surveys by governmental, academic, and private research organizations during various time periods.
For many years, Naikakufu [Cabinet Office] has been sponsoring many public opinion surveys in Japan. The present study examined three Naikakufu-sponsored public opinion surveys conducted regularly between 1997 and 2018. Namely, they were the Kokumin Seikatsu ni kansuru
Yoron Choosa [Public Opinion Survey on the Life of the People]; the Shakai Ishiki ni kansuru Yoron Choosa; [Public Opinion Survey on Social Awareness]; and the Gaikoo ni kansuru Yoron Choosa
[Public Opinion Survey on Diplomacy]. All the raw data were obtained from the annual editions of
Kokumin Seikatsu (Naikakufu, not dated “a”), Shakai Ishiki (Naikakufu, not dated “b”), and Gaikoo
(Naikakufu, not dated “c”) and corresponding printed reports by Naikakufu for these surveys. These three government-sponsored surveys used personal interviews with Japanese of voting age. The individuals were drawn from the Juumin Kihon Daichoo [Basic Resident Registers] using 2-stage probability sampling. The surveys had similar methodologies and used mostly (but not consistently) the same fieldwork companies. All three are large-scale nationwide surveys: The annually drawn sample (during this period) for each of the first two surveys was 10,000, and for Gaikoo it was 3,000. Further, these surveys use the same method of computing and reporting their response rates. Specifically, their response rates are computed from the total number of completed interviews divided by the total number drawn with nothing subtracted from it.
The three surveys were conducted annually (or mostly annually) during this 22-year (1997– 2018) period. Specifically, the Kokumin Seikatsu survey was conducted 20 times; the Shakai Ishiki 19 times; and the Gaikoo 22 times.
Overall response rates
All three surveys had statistically significant declining trends in their overall response rates during 1997–2018. Also, the observed decreases had relatively similar levels of steepness in these surveys.
The three surveys exhibited unusually sharp decreases in their response rates around 2005 and 2006. These decreases can be attributed to the negative publicity about public opinion surveys in the mass media at that time. Fortunately, in subsequent years response rates recovered somewhat from these lows.
A comparison of the first (1997) and last (2018) year of this period showed decreases ranging between 13 and 17 percentage points in the three surveys. That is, response rates declined from 72.9% to 59.9% in Kokumin Seikatsu, from 71.0% to 57.4% in Shakai Ishiki, and from 69.3% to 55.4% in Gaikoo.
Response rates by gender
The response rates of males and females exhibited statistically significant declining trends during 1997–2018. These decreasing trends were of relatively similar magnitudes in the three surveys.
Decreases in the response rates of females were steeper than those of males during these years. However, response rates of females were consistently higher than those of males during 1997–2018. Specifically, the mean differences in the three surveys between the response rates of males and females were between 7 and 8 percentage points during these years.
Response rates by age
Age was classified in five categories for this study. These were: “29 or less,” “30–39,” “40– 49,” “50–59,” and “60 or more.”
In all three surveys, the response rates had statistically significant declining trends for all age categories during this period. Specifically, as age decreased so did the response rates. For all age categories, the three surveys exhibited similar patterns of decreases in their response rates.
The largest differences in response rates were between persons of the oldest and the youngest age groups. That is, comparing those “60 or more” with those “29 or less” revealed that the average
response rates of the latter were between 25 and 26 percentage points lower in the three surveys during this time period.
Response rates by gender and age
The response rates of all male and female age groups showed statistically significant declining trends during 1997–2018. The response rates of females “29 or less” resembled those of males of the same age group. However, response rates of females “29 or less” averaged approximately five percentage points higher than those of males of that age group during this period. In the three surveys, this mean percentage point difference was of the same magnitude.
The patterns of response rates of the various age groups of males and females were somewhat different during this time period. Response rates of males decreased steadily as age category decreased. That is, there were generally relatively clear separations in the response rates among the various male age groups during this period. Except of those “29 or less” and to some extend of those in their thirties, the other female age groups (forties, fifties, and 60 or older) had response rates that were relatively similar to each other.
Response rates by location
The three surveys use the same categories for location. These categories are: Tokyo wards, major cities, middle cities, small cities, and towns & villages.
There were significant declining trends of response rates in all rural and urban areas during 1997–2018. The magnitudes of decreases in response rates by degree of urbanization were relatively similar in the three surveys. Further, the steepest declines for all three surveys were noted in Tokyo wards.
During the period of 1997–2018, the response rates in Tokyo wards averaged between 50% and 52% in the three surveys. During the same period, the average response rates in towns & villages ranged from 67% to 70% in these surveys. That is, in Tokyo wards the average response rates were between 16 and 18 percentage points lower than those of towns & villages. It is noteworthy that the average response rates of towns & villages during this period were somewhat similar to those of small cities with the former being slightly higher.
Reasons of nonresponse
The three surveys use the same nonresponse categories. These are: Refusal, short term absence, could not locate address, moved, long term absence, and other. For this analysis, the last four categories of nonresponse were combined into “other reasons.”
In the three surveys, refusals averaged between 14%–16% of the number drawn and short term absences between 13%–14%. The average rates for “other reasons” were between 8%–9% of the number drawn in the three surveys during this period.
During 1997–2018, the trend of increasing refusal rates was statistically significant in the Kokumin Seikatsu survey. However, changes in the rates of refusals did not reach statistical significance in Shakai Ishiki and Gaikoo during this period.
In all three surveys, the trends of increasing rates of short term absences were statistically significant during 1997–2018. Similarly, the three surveys had significant increases in their trends for “other reasons” of nonresponse.
Summary and closing thoughts
It is important to investigate response rates across the years. Analyzing trends in response rates can assist researchers understand various issues related to nonresponse. In turn, such analyses can help the search for solutions to this major problem of survey research.
There were substantial similarities in the response rate trends of the three surveys that were examined. During 1997–2018, the decreases in overall response rates were generally larger than those noted (Synodinos & Yamada, 2000) for earlier years.
There were large differences in response rates by gender, age, and location. That is: Females had higher rates than males; as age decreased so did the response rates; and with increasing degrees of urbanization response rates decreased. There were some differences between the patterns of the response rate trends of the various male age groups vis-à-vis those of the corresponding female age groups. Generally, the findings regarding these variables (gender, age, location, and gender by age) were very similar in the three surveys. Further, there were mostly similarities (and some differences) among these surveys in the trends of their reasons of nonresponse.
During 1997–2018, refusals were the largest component of nonresponse. This indicates a change from earlier years when short term absences accounted for the largest portion (Synodinos & Yamada, 2000) of nonresponse.
The large declines in response rates around 2005 and 2006 emphasize the importance of avoiding scandals in the survey research industry. Researchers should be mindful of negative publicity about surveys and be prepared to address such occurrences with appropriate measures. A positive image of survey research must be maintained among the general public. Unfortunately, this is not an easy task to accomplish and a single event can have many unforeseen negative consequences.
References
Ando, Masayo (2012). Mensetsu choosa ni okeru kaishuu ritsu koojo no tamenoo ichi koosatsu – Taishoosha anketto, choosain anketto no kekka kara [Considerations for improving the response rates of personal interviews: From survey of respondents and survey of interviewers]. Shakai to Choosa [Advances in Social Research], number 9, 76–89.
Ando, Nanae (2009). Choosa-gaisha no kakaeru kadai [Problems of fieldwork companies]. Shakai to Choosa [Advances in Social Research], number 3, 65–71.
Aramaki, Hiroshi (2010). Yoron choosa no shuhoo ni kansuru genjoo to mondaiten [Recent condition and issues about the methods of public opinion polls]. Massu Komyunikeishon Kenkyuu [Journal of Mass Communication Studies], number 77, 59–75.
Hino, Airo (2015). Nihon ni okeru yoron choosa no genjoo to kadai [Present situation and problems of public opinion surveys in Japan]. Yoron [Public Opinion], number 115, 21–37.
Kikkawa, Tooru (2010). Kyohi zooka ni ikani taioo suru ka? [How to cope with the increases in refusals?]. Shakai to Choosa [Advances in Social Research], number 5, 16–25.
Konuki, Shiroo. (1985). Kaishuu ritsu no dookoo to funoo riyuu no henka [Trends in response rates and changes in the reasons of nonresponse]. Chuuoo Choosa Hoo [Newsletter of Central Research Services], number 327, January (pp. 7–9); number 328, February (pp. 6–7); and number 329, March (pp. 7–9).
Naikakufu Yoron Choosa (not dated “a”). Kokumin Seikatsu ni kansuru Yoron Choosa – Ichiran [Public Opinion Survey on the Life of the People – List]. Last retrieved February 24, 2020 from https://survey.gov-online.go.jp/index-ko.html
Naikakufu Yoron Choosa (not dated “b”). Shakai Ishiki ni kansuru Yoron Choosa – Ichiran [Public Opinion Survey on Social Awareness – List]. Last retrieved February 24, 2020 from https://survey.gov-online.go.jp/index-sha.html
Naikakufu Yoron Choosa (not dated “c”). Gaikoo ni kansuru Yoron Choosa – Ichiran [Public Opinion Survey on Diplomacy – List]. Last retrieved February 24, 2020 from https://survey.gov-online.go.jp/index-gai.html
Onodera, Noriko. (1995). Yoron choosa ni okeru choosa funoo to seido [Nonresponse in opinion polls]. Hoosoo Kenkyuu to Choosa [The NHK Monthly Report on Broadcast Research], 45(9), 54–57.
social surveys]. Shakai to Choosa [Advances in Social Research], number 5, 5–15.
Sugiyama, Meiko. (1992). Choosa funoo to choosa seido [Nonresponse and accuracy of surveys]. Yoron [Public Opinion], number 69, 13–25.
Synodinos, Nicolaos E., & Yamada, Shigeru (2000). Response rate trends in Japanese surveys. International Journal of Public Opinion Research, 12(1), 48–72.
Tsuruta, Tomohisa (2008). Mensetsu choosa no genjoo to kadai [The present conditions and a problem of the face-to-face interview]. Koodoo Keiryoo Gaku [The Japanese Journal of Behaviormetrics], 35(1), 5–16.
Ujiie, Yutaka & Takeshita, Yoshiaki. (1995). Yoron choosa no choosa funoo ni tsuite [Nonresponse in public opinion surveys]. In Nihon Koodoo Keiryoo Gakkai [Behaviormetric Society of Japan] (Ed.). Dai 23 Kai Taikai Happyoo Ronbun Sairokushuu [Collection of papers presented at the 23rd Conference], (pp. 272–275). Kansai University, Suita City, Osaka: Nihon Koodoo Keiryoo Gakkai.
Acknowledgment:
Sincere thanks to the Institute of Statistical Mathematics (Tokyo) where the first author had visiting appointments.
Authors’ information:
Nicolaos E. Synodinos is Professor at the Department of Marketing, Shidler College of Business, University of Hawaii at Manoa, Honolulu, HAWAII 96822, USA.
Prior to his untimely death, Shigeru Yamada was Professor at the Seikei Department, Kokushikan University, Tokyo.
