Deft Research recently conducted a research project uniquely featuring parallel data collection using Internet surveys and telephone interviews. Using the Internet, researchers can obtain information from a large number of respondents quickly and inexpensively, however we don’t know whether Internet research contains biases and whether whatever biases are found are predictable from one project to the next. We are not aware of another project that used both telephone and Internet in side by side research about senior citizen health insurance. This is a unique chance for comparison.

COMPARISON

More detail about the specific study and comparison results is available at
BEAUTIFUL THINGS

Below is a table showing correlations and average differences between Internet and Telephone responses. Where the correlation is high (closer to 1.000), the the distribution of answers over the possible response options was similar for Internet and telephone respondents. The lower the correlation, the less alike are the answers of the two groups. In the one case where the correlation is negative, Internet and telephone respondents tended to answer in opposite ways.

In this analysis, correlations tended to be high, so the next thing to look at is whether there was much difference in responses. The table shows the average difference in response frequency between the two groups. High correlations and smaller differences indicate the two groups responded almost identically. High correlations and larger differences indicate that one of the two groups tended to answer at higher ends of the response option scale than the other; the distribution of their answers had the same shape (bell curve for example) but one group’s distribution was over lower values than another’s. An example in this case, telephone respondents tended to answer more positively to questions about market differences.

STATISTICAL COMPARISONS OF INTERNET AND TELEPHONE RESPONSES

* A high correlation nearing 1.00 indicates that Internet and Telephone responses have an more identically shaped distribution of values along a range of response options. A negative correlation means that where Internet responses are high, Telephone responses are low , and vice versa.

CONCLUSION

Telephone and Internet methods produced different results for some questions and nearly identical results for others. Overall, the methods produce similar distributions along a range of response options, but the values differ on average by 9 percent.

The methods were similar on questions having to do with actual past events such as whether someone helped with a decision and what method of enrollment was used. They were also similar on questions with scaled response options (11 point scales).

In some cases, the response values provided by Internet and Telephone respondents are different even though the “shape” of the response distribution was the same. In these cases, we would say the two methods produce “parallel” results.

The methods are most different when questions involve perception or sensitive information. Telephone respondents were more likely to provide a positive response to questions about perceived market differences. Internet respondents, using a more private media, were more likely to indicate they couldn’t afford insurance.

Method Note: for this study, telephone respondents were not given the “Don’t Know” response option, but could answer that way if they thought of it on their own. Internet respondents could see all response options including “Don’t Know” laid out before them. We think this is a partial reason for the tendency of telephone respondents to give more positive answers.

These results can be used by Deft Researchers and others when constructing future questionnaires.

Leave a Reply

You must be logged in to post a comment.