PROBLEM (ETHICS)

POSSIBLE SOLUTION

Respondents may wish to keep their identity from the researcher, and an e-mail address identifies the respondent (in the case of sensitive research, e.g. on child abuse or drug abuse, this may involve criminal proceedings if the identity of the respondent is known or able to be tracked by criminal investigators who break into the site). Non-traceability of respondents may be problematic

Direct respondents to a web site rather than to using e-mail correspondence. Provide advice on using non-traceable connections to access and return the survey (e.g. an Internet café, a library, a university). Advise the respondent to print off the survey and return it by post to a given address. Avoid asking respondents to enter a password or to give an e-mail address. Prevent access to unprotected directories and confidential data

Respondents may not know anything about the researcher, or if it is a bona fide piece of research and not simply a marketing ploy

Include the researcher’s affiliation (e.g. University), with a logo if possible.

Informed consent

Ensure that it is easy for respondents to withdraw at any time (e.g. include a ‘Withdraw’ button at the foot of each screen)

PROBLEM (TECHNICAL: HARDWARE AND SOFTWARE)

POSSIBLE SOLUTION

The configuration of the questionnaire may vary from one machine to another (because of web browsers, connection, hardware, software) and can lead to dropout

Opt for simplicity. Test the survey on different computer systems/browsers to ensure consistency. Avoid surveys that require real time completion

The screen as set out by the survey designer may not appear the same as that which appears on the respondent’s screen

Opt for simplicity. Use a commercial survey software system for generating the questionnaire. Avoid high level programs

Slow network connections or limited bandwidth can slow done loading

Keep the use of graphics to a minimum. Advise on the possible time it takes to load

Respondents may not have the same software, or the same version of the software as the sender, rendering downloading of the questionnaire either impossible or distorting the received graphics

Avoid the use of graphics and more advanced software programs

Graphics may be corrupted/incompatible between the sender and the user, i.e. between one kind of machine, user platform and software and another. Hardware may differ between sender and receiver

Opt for simplicity. Use commercially available web-based surveying systems and packages. Use image files (e.g. .jpeg, .gif) to reduce loading time. Avoid pop-ups if possible as they reduce response rate

The greater the use of graphics and plug-ins (e.g. using Java and Applets), the longer it takes to download, and, particularly – though not exclusively – if respondents do not have broadband access then time-consuming downloads could result in either the respondent giving up and cancelling the download, or creating a bad mood in the respondent

Keep software requirements as low-tech as possible. Avoid questionnaires that use sophisticated computer graphics

There may be slow loading times due to Internet congestion

Avoid sophisticated graphics and ‘fancy’ presentations as these take longer to download

The physical distance between points on an attitude scale may spread out because of configuration differences between machines

Indicate how best the questionnaire may be viewed (e.g. 800 x 400)

The construction procedures for wrap-around text may vary between computers

Keep lines of text short

E-mail questionnaires may distort the layout of the questionnaire (some e-mail software uses HTML, others do not);

Avoid sending a questionnaire directly using e-mail; rather, post it on a web site (e.g. so that respondents visit a web site and then click a box for immediate transfer to the questionnaire). Consider using an e-mail to direct participants to a web site (e.g. the e-mail includes the web-site which can be reached by clicking in the address contained in the e-mail). Use an e-mail that includes an attachment which contains the more graphically sophisticated survey instrument itself

PROBLEM (RESPONDENTS)

POSSIBLE SOLUTION

Respondents may be unfamiliar or inexperienced with the Internet and the media

Keep the questionnaire simple and easy to complete

Respondents may send multiple copies of their completed questionnaire from the same or different addresses

Have a security device that tracks and limits (as far as possible) respondents who may be returning the same questionnaire on more than one occasion. Use passwords (though this, itself, may create problems of identifiability). Collect personal identification items. Check for internal consistency across submissions

There may be more than one respondent to a single questionnaire (the same problem as in, for example, a postal questionnaire)

Include questions to cross-check the consistency of replies to similar items.

Respondents may not be used to pull-down menus

Provide clear instructions

The language of e-mail surveys can risk offending potential participants (‘flaming’)

Check the language used to avoid angering the participants

Respondents’ difficulty in navigating the pages of the online survey

Keep instructions to the page in question. Make the instructions for branching very clear (font size, colour etc.)

PROBLEM (LAYOUT AND PRESENTATION)

POSSIBLE SOLUTION

A page of paper is longer than it is wide, but a screen is wider than it is long, and a screen is smaller than a page, i.e. layout becomes a matter of concern

Remember that screen-based surveys take a greater number of screens than their equivalent number of pages in a paper copy. Sectionalise the questionnaire so that each section fills the screen, and does not take more than one screen

The layout of the text and instructions assumes greater importance than for paper questionnaires

Opt for clarity and simplicity

The layout uses a lot of grids and matrices

Avoid grids and matrices: they are a major source of non-response

The order of items affects response rates

Locate requests for personal information at the beginning of the survey. Include ‘warm-ups’ and early ‘high hurdles’ to avoid dropout

Respondents may be bombarded with too much information in an introductory message

Place the advertisement for the survey on user groups as well as the general public, inviting participants to contact such-and-such a person or web site for further information and the questionnaire itself, i.e. separate the questionnaire from the advertisement for/ introduction to the questionnaire

Respondents may be overloaded with instructions at the beginning of the survey

Avoid placing all the instructions at the start of the questionnaire, but keep specific instructions for specific questions

Respondents may be overloaded with information at the beginning of the survey

Keep the initial information brief and embed further information deeper in the survey

Respondents may have to take multiple actions in order to answer each question (e.g. clicking on an answer, moving the scroll bar, clicking for the next screen, clicking to submit a screen of information

Keep the number of actions required in order to move on to a minimum

Respondents may not be able to see all the option choices without scrolling down the screen

Ensure that the whole item and options are contained on a single screen

Respondents may not understand instructions

Provide a helpline, e-mail address or contact details of the researcher. Pilot the instrument.

Instructions about options may be unclear

Use radio buttons for single choice items, and try to keep layout similar to a paper layout

Respondents only read part of each question before going to the response category

Keep instructions and words to a necessary minimum.

PROBLEM (RELIABILITY)

POSSIBLE SOLUTION

Respondents may alter the instrument itself. The researcher relinquishes a greater amount of control to the respondents than in conventional questionnaires

Include technological safeguards to prevent alteration and have procedures to identify altered instruments

Respondents may be forced to answer every question even when they consider some response categories inappropriate

Pilot the survey. Include options such as ‘don’t know’ and ‘do not wish to answer’ and avoid forcing respondents to reply before they can move on

Respondents may not be telling the truth – they may misrepresent themselves

Include questions to cross-check replies (to try to reduce the problem of respondents not telling the truth)

PROBLEM (DROPOUT)

POSSIBLE SOLUTION

Respondents may lose interest after a while and abandon the survey, thereby losing all the survey data

Have a device that requires respondents to send their replies screen by screen (e.g. a ‘Submit’ button at the foot of each screen) section by section, or item by item. Put each question or each section on a separate screen, with ‘submit’ at the end of each screen. Adopt a ‘one-item-one-screen’ technique

Respondents may not know how long the questionnaire is, and so may lose interest

Include a device for indicating how far through the questionnaire the respondent has reached: a progress bar at the bottom or the side of the survey

Internet surveys take longer to complete than paper-based surveys

Keep the Internet survey as short, clear and easy to complete as possible.

People do not want to take part, and it is easier for someone to quit or cancel an Internet-based survey than a paper based survey (simply a click of a button)

Increase incentives to participate (e.g. financial incentives, lottery tickets (if they are permitted in the country))

Diminishing returns (the survey response drops off quite quickly). Newsgroup postings and electronic discussion group data are removed, relegated or archived after a period of time (e.g. a week), and readers do not read lower down the lists of postings

Ensure that the web site is re-posted each week during the data collection period

Non-participation may be high (i.e. potential participants may not choose to start, in contrast to those who start and who subsequently drop out)

Increase incentives to participate. Locate personal informational questions at the start of the survey

Error messages (e.g. if an item has not been completed) cause frustration and may cause respondents to abandon the questionnaire

Avoid error messages if possible, but, if not possible, provide clear reasons why the error was made and how to rectify it