WPS8305 Policy Research Working Paper 8305 Impact of Phone Reminders on Survey Response Rates Evidence from a Web-Based Survey in an International Organization Lodewijk Smets Independent Evaluation Group January 2018 Policy Research Working Paper 8305 Abstract This research note investigates the impact of phone remind- percent), the estimated average effect of treatment-on-the- ers on response rates in the context of a web-based survey in treated is even larger, corresponding to an increase of 64 an international organization, the World Bank. After ran- percentage points. Therefore, if ways can be found to increase domly assigning treatment to 248 survey participants, the treatment compliance, high response rates are attainable. study finds an intention-to-treat effect of 19.86 percentage This may lead World Bank surveyors to turn to sample sur- points. Given a relatively low treatment compliance rate (31 veys more often, reducing survey overload in the institution. This paper is a product of the Independent Evaluation Group . It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at lodewijk. smets@gmail.com The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Impact of Phone Reminders on Survey Response Rates: Evidence from a Web-Based Survey in an International Organization Lodewijk Smets1,2 1 The World Bank, Independent Evaluation Group 2 KU Leuven, Centre for Institutions and Economic Performance Keywords: Survey, international organizations, experiment, response rate JEL: C83, C81, C9. 1. Introduction As response rates are related to the risk of non-response bias,1 several techniques have been developed to increase survey response rates, ranging from monetary incentives to personalized invitations.2 Also telephone reminders are used as a way to boost response rates. A number of studies have investigated the impact of phone reminders on survey response rates, at least dating back to Roscoe et al (1975). Somewhat surprisingly however, all studies are conducted in the context of postal surveys. This research note aims to complement this field of study by investigating the impact of phone reminders in the context of a web-survey, targeting professional staff in an international organization, i.e., the World Bank. The present paper is related to the work of Smets and Bogetić (2018) who provide a detailed discussion of the underlying survey and its design, methodology and implementation. To the best of our knowledge, this is the first study to examine survey response rates in such a setting. We randomly assigned the reminder treatment to 248 survey participants. During the telephone call we briefly explained the purpose of the reminder, verified whether participants already filled out the survey and offered to resend the survey link immediately after the call in case they had not.                                                              1  Response rates set bounds to the maximal non-response bias. However, it is important to note that also other factors such as measurement error, coverage error or editing error may bias survey estimates (e.g., see Davern, 2013; Meyer et al, 2015). 2 See Edwards et al (2009) or Fan and Yan (2010) for a recent systematic review of the methods to increase response rates in postal and web-based surveys. 2    This strategy led to an overall response rate in the treatment group of 54.03 percent. The response rate in the control group was 34.17 percent, resulting in an Intention-To-Treat (ITT) effect of 19.86 percentage points. Due to the fact that out of the 248 individuals assigned to treatment, only 77 staff answered the phone call, had not yet filled out the survey but were willing to receive another survey link, the local average treatment effect (LATE) is even larger. The LATE – which in this case measures the average effect of treatment-on-the-treated (TOT) – corresponds to a vast increase of 64 percentage points. The remainder of the note is structured as follows. In the next section we highlight the specificities of conducting a web-based survey in an international organization, discuss the survey design and elaborate on the method used to evaluate the impact of phone reminders on survey response rates. In section 3 we discuss the results while section 4 concludes. 2. Background and Method As part of a larger evaluation, the World Bank’s Independent Evaluation Group (IEG) implemented a web-based survey among professional staff to elicit their views on one of the Bank’s corporate goals. The survey was sent out to a sample of 3,698 eligible staff members.3 It was launched on September 7, 2016 and closed on October 18, 2016. Surveying professional staff in an international organization comes with a number of challenges.4 First, the World Bank Group has more than 130 offices worldwide, which complicates the timing of the survey and the telephone reminders. Relatedly, given its                                                              3  We originally selected a sample of 3,736 units. However, 32 individuals were excluded as they had left the institution by the time the survey was implemented. We also excluded 6 IEG staff members from the sample who were mistakenly included (IEG was not part of the target population), leading to a sample of 3,698 eligible units.  4 One important advantage of a web-based survey targeting staff of an international organization is the availability of a nearly complete sampling frame, thus minimizing coverage bias. 3    global scope, staff travel regularly. From July 2015 until June 2016, 56,090 World Bank flights were booked originating from Headquarters (Washington, D.C.), with top destinations all over the map, from Johannesburg to Kathmandu. Adding another 11,086 trips originating from country offices, the average number of World Bank flights per week amounts to 1,291. Third, web-based surveys are frequently used at the institution to collect data on a wide range of issues – from the quality of commuter services to workforce perspectives on managerial effectiveness. For instance, since 2009 IEG alone sent out 208 web-based surveys, corresponding to an average of 23 surveys per year.5 As Porter et al (2004) note, overexposure to surveys may lead to survey fatigue and declining response rates. Indeed, World Bank survey response rates are generally low and tend to be declining over time. For example, since 1999 IEG has surveyed its clients – WBG staff, the Board of Directors and external stakeholders – annually to gauge their perceptions of the quality and impact of IEG evaluations. In 2006, 22 percent of WBG staff responded to the client survey. In 2012 the response rate dropped to 11.3 percent while the 2015 client survey response rate plunged to a dramatic 4.7 percent. Given these negative trends, it was found important to increase the response rate for this survey. Therefore, IEG’s Director General sent out the initial email invitation, including the expected duration of the survey, the survey deadline and a web link to the survey at the bottom of the invitation. Two reminder emails were sent out, one three weeks after launching the survey and another one the day before closing the survey.                                                              5 This includes both internal surveys targeting IEG staff as well as external surveys targeting the World Bank Group and other stakeholders (e.g., client governments). 4    Between the two reminders – on October 12-13 – we called a random subset of the sample inviting them to take the survey in case they had not. During the telephone call we briefly explained the purpose of the reminder, verified whether participants already filled out the survey and offered to resend the survey link immediately after the call in case they had not. Of the survey participants, 248 were randomly assigned to receive the treatment, while the remaining 3,450 participants served as the control group. As shows, there is no significant treatment-control imbalance on four observable characteristics – organizational mapping, pay grade, geographic location and managerial status – for which we have information. Furthermore, to verify joint orthogonality, we also regressed the treatment assignment on the four covariates. The F-test of this regression is 0.67 (p-value= 0.611), indicating that jointly the covariates do not explain treatment assignment. Table 1: Balancing tests covariate mean treatment mean control p-value IBRD 0.725807 0.72058 0.859 Grade 1.895161 1.936232 0.423 Location 0.596774 0.557101 0.224 Manager 0.060484 0.074203 0.392 Note: The first column lists the four covariates on which we have information. IBRD is a dummy variable that stands for the organizational mapping of the sample unit. It equals 1 if the sampled individual belongs to the International Bank for Reconstruction and Development (IBRD), 0 otherwise. Grade is a categorical variable indicating pay grade. Location is a dummy variable equal to 1 in case the sampled individual is located at Headquarters (Washington D.C.), 0 otherwise. And finally Manager is a dummy equal to 1 in case the sampled individual is a manager. The second and third column represent the mean values for the treatment and control group respectively. The forth column represents the p-value on treatment in an OLS regression of the covariate on assignment to treatment. 5    Of the 248 individuals assigned to treatment, only 77 staff answered the phone call, had not yet filled out the survey but were willing to receive another survey link.6 Following the notation in Angrist and Pischke (2009), we estimate the intention-to-treat effect (ITT) as: | 1 | 0 With Yi a binary variable equal to 1 in case the sampled individual participated in the survey and zi a binary variable equal to 1 in case individual i was assigned to treatment.7 We can use the random assignment to estimate the effect of treatment on the group of “compliers”.8 Since we are faced with one-sided non-compliance (the control group did not have access to the treatment), this effect is equal to the average effect of treatment- on-the-treated (TOT).9 It can be estimated as: | 1 | 0 | 1 1| 1 With Y1i, Y0i individual i’s potential outcomes under treatment and control respectively, Di treatment status and 1| 1 the treatment compliance rate.                                                              6 It was considered important to guarantee full anonymity when implementing the survey. As a consequence, we did not know beforehand whether or not the sampled staff members completed the survey. Out of the 110 people we were able to reach, 29 already had and 81 had not. Out of the 81 that had not completed the survey, 77 agreed to receive another email with the survey link. Out of those 77 people, 55 participated in the survey. 7 Our outcome measure also counts partial interviews as respondents. As such the response rate corresponds to AAPOR’s RR2 definition. 8 Since the group that actually complies with the assigned treatment is self-selected, a comparison between that group and the control is again subject to selection bias. Using the (random) treatment assignment as an instrument solves this compliance problem. 9 The following key identifying assumptions need to be met: conditional independence of the instrument and monotonicity (see Imbens and Angrist (1994) for more detail). For this experiment, this is highly likely to be the case. 6    3. Results In the control group 1,179 individuals participated in the survey, corresponding to a response rate of 34.17 percent (=1,179/3,450). In the treatment group 134 individuals participated in the survey,10 resulting in a response rate of 54.03 percent (=134/248). Thus, the ITT effect is 19.86 percentage points (p-value=.00001).11 Given the relatively low treatment compliance rate (=77/248=0.31), the average effect of treatment-on-the-treated is more than three times as large: 248 0.1986 ∗ 0.6396 77 Therefore, if ways can be found to increase treatment compliance, (very) high response rates are within reach. In the context of this survey, we see three ways of increasing treatment compliance. First, due to time constraints, we were only able to make two telephone attempts. Increasing the number of calls is likely to improve treatment compliance (Munoz-Leiva et al, 2010). Second, knowing when staff are out-of-office can help in optimizing the timing of the telephone reminders. Finally, the survey was implemented with full anonymity, so that we did not know beforehand whether or not staff members completed the survey. Therefore, giving up on anonymity leads to better targeting of the phone reminders (even though it could potentially decrease survey participation).                                                              10 Given full anonymity and imperfect compliance, we had to estimate the response rate of the 138 “non-compliers” in the treatment group. We used the overall response rate at the start of the experiment and the change in the number of responses from the start of the experiment until survey closure to do so. 11 P-value from employing a z-test for differences in proportions estimated from two independent samples. 7    4. Discussion Results from this experiment show that phone reminders can be a cost-effective way to increase survey response rates. That is, 55 out of the 110 people we were able to reach ended up filling out the survey. Given a labor cost of $200 per day – and two days of calling the treatment group – the cost per additional respondent is $7.27.12 What is more, given the large TOT effect, (very) high response rates are within reach. This opens up the possibility of conducting sample surveys more regularly instead of population surveys. That is, because of low response rates for World Bank surveys, sampling does not guarantee representativeness. Furthermore, since the financial cost of sending out the survey to the whole population is low, such an approach is often taken. But this strategy does not internalize externalities such as survey fatigue and (high) opportunity costs of filling out questionnaires.13 Knowing that high response rates are attainable, World Bank surveyors may turn to sample surveys more often, therefore reducing survey overload. While these results have high internal validity, caution is required when generalizing the findings to other situations/surveys. Some important situational variables need to be taken into account. First, the survey topic was about one of the World Bank’s corporate goals. This may have raised additional interest from survey participants. Second, the timing of the survey was specifically chosen as to ensure staff availability. Third, the majority of World Bank staff are proud to work at the WBG and are motivated to contribute more than what is expected (World Bank, 2016). Such personal characteristics – which are                                                              12 However, a more effective targeting strategy may further increase the cost-effectiveness of the phone reminders. 13 In an email conversation with an IEG Director, the World Bank’s Chief Economist raised the issue of high opportunity costs. 8    arguably related to the propensity to respond – may not necessarily translate to other institutions, hence limiting external validity. For these reasons we believe that findings from other international institutions as well as evidence from other surveys within the WBG may provide valuable information. 9    References Angrist, J. D. and Pischke, J-S., 2009. Mostly Harmless Econometrics: An Empiricist's Companion. Princeton University Press. Davern, M., 2013. Nonresponse Rates Are a Problematic Indicator of Nonresponse Bias in Survey Research. Health Services Research 48 (3): 905-912. Edwards, P.J., Roberts, I., Clarke, M.J., DiGuiseppi, C., Wentz, R., Kwan, I., Cooper, R., Felix, L.M., Pratap, S., 2009. Methods to increase response to postal and electronic questionnaires. Cochrane Database of Systematic Reviews 2009, Issue 3. Fan, W. and Yan, Z., 2010. Factors affecting response rates of the web-survey: a systematic review. Computers in Human Behavior 26: 132-139. Imbens, G. W. and Angrist, J. D., 1994. Identification and Estimation of Local Average Treatment Effects. Econometrica 62 (2): 467-475. Meyer, B. D., Mok, W. K. C. and Sullivan, J. X. 2015. Household Surveys in Crisis. Journal of Economic Perspectives 29 (4): 199-226. Munoz-Leiva, F., Sanchez-Fernandez, J., Montoro-Rios, F. and Ibanez-Zapata, J.A., 2010. Improving response rate and quality in Web-based surveys through personalization and frequency of reminder mailings. Quality and Quantity 44: 1037-1052. Porter, S. R., Whitcomb, M. E. and Weitzer, W. H., 2004. Multiple surveys of students and survey fatigue. New Directions for Institutional Research, 2004: 63–73. Roscoe, A.M., Lang, D., Sheth, J.N., 1975. Follow-up methods, questionnaire length, and market differences in mail surveys. Journal of Marketing 39: 20–27. Smets, L. and Ž. Bogetić (2018). An Evaluative Look Behind the Curtain: World Bank Group Staff’s Early Experience with the Shared Prosperity Goal. World Bank Policy Research Working Paper 8293, Washington, D.C. World Bank, 2016. World Bank Group Engagement Survey. World Bank, Washington, D.C. 10