THE EFFECT OF CREDENTIALS AND INCENTIVES ON RESPONSE RATE FROM A SAMPLE POPULATION OF ENGINEERS

Authors: William S. Lightfoot & Lydia Porter

The use of self-administered questionnaires for gathering data from a sample group has well-documented tools and measures of efficacy (Aikens 1997, Dillman 2000). The process of delivering self-administered surveys to potential respondents is equally refined, as are the means for making the questionnaires as valid and reliable as possible. From construct, to response rate, to data analysis, there are systematic approaches that are widely established and accepted.
In spite of this, survey non-response continues to be a significant issue for researchers. The lower the response rate, the higher the non-response, and the increased likelihood that the results will not be representative of the population as a whole. Survey response rate is therefore a critical factor in increasing the external validity of a research study.

In attempting to increase response rate, a survey might include factors such as advance notification, incentives, and personalization of correspondence, with varying levels of success (Dillman 2000, Mehta & Sivadas 1995, Witt & Poytner 1997). Groves and Couper (1998) presented a framework for understanding the factors that contribute to survey participation and response. They indicate that social environment, survey design, respondent environment and interviewer characteristics feed into the interaction between the respondent and the interviewer, all of which lead to a decision by the respondent on whether or not to cooperate. In addition, rewards, cost, and trust are three critical elements necessary for predicting a specific action on the part of potential respondents (Dillman, 2000). Unfortunately, the effect of such variables as credentialing, incentives, and contact strategy remain largely unknown (Fine, 2000) or seem to have no productive effect on improving response rate (Dennis, 2003).

Credentials and Sponsorship
Often, the first thing potential respondents see when they receive a survey package is a logo displaying the researcher's credentials, or that of the sponsoring client or organization. When this is printed on the envelope, it may serve as the initial impression by which a respondent begins to decide whether or not to respond to the survey. If participants recognize the credentials of a researcher, it may positively affect the response rate, as suggested by Donald (1960) who states that people who are more involved with the sponsor of the survey will tend to have a lower rate of non-response. The view that participants may be more inclined to complete a survey if an authority or legitimate sponsoring organization is involved, is also supported by Kanetkar (2000), although some individuals who do not feel part of the larger society, may take a negative view and refuse to participate. The evidence seems to support the fact that the prominent display of a sponsor's brand or logo can often reassure respondents (Virtual Surveys 2001), and as a consequence may have an impact on response rate.

There seems to be conflicting evidence as to the benefits of using one type of credential over another, such as academic sponsorship in preference to corporate sponsorship. Numerous articles and studies have indicated that academic sponsorship has a positive effect on response rate, and therefore non-response (Bush & Burns, 2000; Dillman, 2000). In addition, some studies have found that university sponsorship results in higher response rates than corporate sponsorship (Fox Crask & Kim 1988, Goyder 1982, Heberlein & Baumgartner 1978, Larson & Poist 2004), although these positive effects may be limited geographically (Jones 1979). Other researchers suggest there is no significant difference between response rates for sponsoring organizations (Brunner & Carroll, 1969, Jones & Linda, 1978) or that it is response distribution that is affected by sponsorship (Presser, Blair & Triplett 1992)

Pradeep (1989) noted that social utility appeals resulted in higher response rates when the sponsor was an academic institution, whereas appeals that emphasized the importance of the respondent's individual responses resulted in higher response rates when the sponsoring organization was a commercial firm.

Incentives
Many researchers try to increase response rates by using incentives such as coins or dollar bills, lottery tickets, offers of entry into a drawing, and discount coupons. Incentives that reward participants after completion of the survey have little apparent effect on response rate, in contrast to incentives that reward potential respondents immediately (Dillman 1978). These findings are supported by Church (1993), who showed that "token" incentives when given with a request to complete a questionnaire, consistently improve response rates, while promises to pay people after they have completed the questionnaire, do not. If an incentive is offered, the respondent may be inclined to take the survey simply because of the incentive in an attempt to obtain the reward (Dillman 2000). More recent research by Jobber, Saunders & Mitchell (2004) confirms that monetary incentives tend to increase response rate, and suggests that the value of the incentive may also affect the response.

Research Study
This study aims to investigate the effects of credentials, and incentives on response rates for direct mail delivered, self-administered surveys.

Two of the hypotheses tested in this research were:
H1: Credentials do not affect response rate.
H2: Incentives do not produce a higher response rate than non-incentives.

The two independent variables were incentives and credentials, and the dependent variable to be evaluated was the response rate.

Sample Size
The population used in this study included engineers involved in the design, specification, and selection of motors, drives, and motion control. The sample was drawn from a list of subscribers to a leading trade magazine, all of whom had self selected direct mail as a contact method for receiving information.

Based on input from a client, it was assumed that this database was representative of the best target market for the survey and included only subscribers to a single trade publication. In addition, a survey was assumed to be the best method for gathering the information in this context. Potential survey participants were selected based solely on how they chose to be contacted by the magazine's publisher. A total of 7,735 subscribers fit the demographic criteria for participation in the survey and had specifically requested that they be contacted for special offers and information via direct mail delivery. From this list, 3000 names were randomly selected using systematic sampling, whereby the list owner's personnel chose every fifth name until the list was complete.

According to the publisher (Cahners, 2000), the demographic profile of this list is as follows:
1. 68.3% Specify and/or buy Motors/Drives
2. 46% hold Electrical Engineering degrees
3. 16% hold Mechanical Engineering degrees
4. 11% hold Chemical Engineering degrees
5. 24% hold degrees in Science & Mathematics
6. 8% hold other degrees

Procedure
The survey was developed based on a previous one used by the trade magazine, with input from the client, and was adapted to be used on a single side of a tri-fold mailer, the back which was pre-addressed and stamped to make it easier for the target population to respond. Surveys were mailed to 2400 participants using one of three different cover letters stating either the college, marketing, or consulting credentials of the researcher. (The researcher worked for each of the three organizations who partially sponsored this research.) One of three different incentives were identified on the survey itself, and within each initial mailing of 800, 350 surveys were sent with no incentive, 400 surveys offered to put respondents' names into a drawing for a chance to win $250, and 50 surveys contained a $1 coin attached to the survey.

It was decided that due to budgetary constraints, one initial mailing and one follow up mailing would be performed.

Data Analysis
The analysis of the data was in two phases. First, an initial analysis of the response time and rates was performed to enable comparison with other studies. Second, the hypotheses were tested using Chi-Square tests for response rate.

Initial Analysis
The mean response time was 15.38 days +/- 14.39 days, which is consistent with past experiments that have measured mean response time. A total of 2313 surveys were delivered, with 280 usable responses, which indicates that the researcher can be 95% certain that the response distributions of the general population are within +/- 5.75% of the actual responses. An overall response rate of 12.1% was achieved, calculated as the net number of usable responses after returned, and unusable responses were accounted for. Responses were counted if at least 80% of the questions were answered, and if the respondent completed all parts consistently of questions 1 and 11 (see Figure 1). Further analysis compared the response rates for surveys with different credentials and incentives, the results of which are shown in Table 1.

TEST OF HYPOTHESIS
Test of Hypothesis 1: Effect of Credentials on response rate.
H1 states that credentials do not affect response rate.
To test the response rates, a chi-square analysis was performed testing the effect of credentials (Table 2). The test level of statistical significance was selected as a = .05. The results of the chi-square test indicate no support for H1, X2 = 6.65, p <.05.

Test of Hypothesis 2: Effect of Incentives on response rate.
H2 stated that incentives do not produce a higher response rate than non-incentives.
To test the response rates, a chi-square analysis was performed testing delivery media. The test level of statistical significance was selected as a = .05. The results of the chi-square test indicate no support for H2, X2 = 41.99, p <.05.

Discussion of Results
There were some limiting factors that should be considered in a discussion of the results of the study. Financial constraints necessarily kept the sample size smaller than would have been preferred. The reward mechanism was also a limitation, as direct mail surveys often use a direct financial incentive, which has been shown to increase response rate. Finally, the sampling frame error is unknown as the owner of the mailing list selected and managed the mailing lists used in this study.

In spite of this, an overall response rate of between 10 and 20% was expected based on past experience, and for similar surveys, a range of between 10 and 30% seems quite reasonable for a population drawn from such a technical audience (Appliance Manufacturer, 2000). The findings of this survey are therefore, generally consistent with those from other research projects.
Nevertheless, non-response is clearly an issue, as the majority of potential respondents chose not to respond to the surveys. The client needs to be aware that the data is not necessarily representative of the entire population. Further analysis may find that the sample sizes were too small relative to the population, which may have led to data sets drawn from slightly different samples within the population. Perhaps a higher response rate, with more samples would yield a different result.

The response rates in this study were low enough that the samples may reflect sub-sets of the larger population of subscribers to the trade publication. This indicates that the data may have a high degree of error, and may not be useful from a purely statistical point of view. The results may lead to further analysis of the market, and the identification of specific sub-sets within the overall population that should either be avoided, or might be contacted again for future surveys.

Effect of Credentials
The analysis indicates that researcher credentials have a significant effect on survey response rate. Specifically, the response rate for direct mail surveys, which used college sponsorship and credentials (14.4% overall, including follow up mailings) yielded a higher response rate than either the consulting (11.8%), or marketing firm (9.5%) sponsored surveys. This is consistent with other studies (Bush & Burns, 2000; Dillman, 2000; Larson & Poist, 2004).

Effect of Incentives
The results indicate that a direct financial incentive can significantly increase response rate, while the promise of an incentive has a minimal effect (see Table 1). This is also consistent with past research that has shown that a direct financial incentive such as a coin, or dollar bill substantially increases response rate (Dillman 1978). There is some debate as to the magnitude of the increase, with at least one article indicating that a higher incentive may in fact result in a lower response rate than a smaller incentive (National Computer Systems, 1997) and a more recent study suggesting that response rates would increase as the value of the incentive increases (Jobber, Saunders & Mitchell, 2004). It is suggested that the recipient perceives a higher incentive as payment for services rendered, rather than provoking the more desirable obligation to complete the survey.

Recommendations for Future Research
This study found that there is a difference in response rate based on credentials of the researcher, or incentives. However, many different variables affect response rate and corresponding sample size. Survey length, survey design, and saliency could also be relevant and are worth exploring further as they may help explain why the non-response was higher than desired. The tests of credentials and incentives also found significant differences that support the current state of knowledge about direct mail delivered surveys. Further evaluation of these results may provide additional insight into the relationships between credentials, incentives, and response rate.

References

1. Aikens, L. R. (1997) Questionnaires and Inventories: Surveying Opinions and Assessing Personality. New York: John Wiley & Sons.
   
2. Appliance Manufacturer, Reader Preference Study, August 2000. Business News Publishing Company.
   
3. Brunner, G. A. & Carroll, S. J. (1969). Weekday Evening Interviews of Employed Persons Are Better. Public Opinion Quarterly, 33 (2), p. 265-267
   
4. Bush, R. F. & Burns, A. C. (2000) Marketing Research (3rd Ed.) Upper Saddle River, N.J.: Prentice-Hall.
   
5. Cahners (2000). Cahners Business Lists. (http://www.cahners.com)
   
6. Church, A. H. (1993). Estimating the effect of incentives on mail survey response rates: a meta-analysis. Public Opinion Quarterly, 57, p. 66-79.
   
7. Dennis, W. J. (2003). Raising response rates in mail surveys of small business owners: results of an experiment. Journal of Small Business Management, 41(3), p. 278-295.
   
8. Dillman, D. A. (1978). Mail and telephone surveys: the Total Design Method. New York: John Wiley & Sons.
   
9. Dillman, D. A. (2000). Mail and telephone surveys: the Total Design Method. New York: John Wiley & Sons.
   
10. Donald, M. N. (1960). Implications of nonresponse for the interpretation of mail Questionnaire data. Public Opinion Quarterly, 14, p. 99-114.
    
11. Fine, B. (2000) Internet Research: The Brave New World. In Chuck Chakrapani (Editor), Marketing Research. State-of-the-Art Perspectives. (pp. 143- 155). Chicago: American Marketing Association.
    
12. Fox, R., Crask, M.R. and Kim, J. (1988) Mail Survey Response Rates: A Meta-Analysis of Selected Techniques for Inducing Response. Public Opinion Quarterly 52, 4 p. 467-91.
    
13. Goyder, J. C. (1982) Further Evidence on Factors Affecting Response Rates to Mailed Questionnaires. American Sociological Review 47, 4, p. 550-53.
    
14. Groves, R. M. & Couper, M. P. (1998). Nonresponse in Household Surveys. New York: John Wiley & Sons.
    
15. Heberlein, T. A. & Baumgartner. R. M. (1978). Factors Affecting Response Rates to Mailed Questionnaires: A Quantitative Analysis of the Published Literature. American Sociological Review, 43, (4), p. 447-462.
    
16. Jobber, D., Saunders, J. & Mitchell, V.-W. (2004). Prepaid monetary incentive effects on mail survey response. Journal of Business Research, April, 57(4), p. 347-351.
    
17. Jones, W. H. (1979) Generalizing mail survey inducement methods: Population interactions with anonymity and sponsorship. Public Opinion Quarterly. 43(1): p. 102-111
    
18. Jones, W.H. & Linda, G. (1978) Multiple criteria effects in a mail survey experiment. Journal of Marketing Research, 1978, 15, p. 280-284.
    
19. Kanetkar, V. (2000) Data Collection Methods and Marketing Research: A Comparison and Review of Alternatives. Chapter 5 (p. 106 - 142) in Marketing Research: State of the Art Perspectives. C. Chakarapani, Ed. Chicago: American Marketing Association.
    
20. Larson, P. D. & Poist R.F. (2004) Improving response rates to mail surveys : a research note. Transportation Journal, Fall, 43(4), p. 67-74.
    
21. National Computer Systems (1997) (http://www.ncs.com.ncscrop/research/ 9701.htm)
    Pradeep, K. T. (1989) The Effects of Appeals, Anonymity, and Feedback on Mail Survey Response Patterns from Salespeople. Journal of the Academy of Marketing Science, 17, (3) p. 235 - 241.
    
22. Presser, S., Blair, J., & Triplett, T. (1992) Survey sponsorship, response rates, and response effects. Social Science Quarterly. 73 (3): p. 699-702.

<< back to Past Issues