Online Surveys: Increasing Your Research and Evaluation Capacity

Abstract: Online surveys are increasingly viewed as a method to expand research and evaluation capacity. This article focuses on a case study utilizing online surveys for data collection conducted by researchers at the Pennsylvania State University and the University of Arizona. The National Youth Participation Study makes use of computer technology and the interest youth have in such technology. Online methods, survey design, techniques when surveying youth and lessons learned in developing the online study are shared by the authors.


Introduction

Today there is a growing demand for research and evaluation data that pertains to young people and the ways they use their time (Carnegie Corporation of New York, 1992; Eccles & Gootman, 2002; Larson, 2000; The William T. Grant Corporation, 2000). Online surveys are increasingly being seen as a method for collecting data and can provide easy access for those participating in the study (Shields, 2003).

This article focuses on a case study utilizing online surveys for data collection. T he National Youth Participation Study, conducted by researchers at the Pennsylvania State University and the University of Arizona, is designed to make use of computer technology and   the interest youth have in such technology. This study was developed as a national online survey of middle school and high school youth (ages 12–18) to examine why they participate in structured activities and programs, such as band, debate team, sports, and 4-H clubs during the out-of-school time hours. The study’s goal is to tap youth’s points of view about what makes programs appealing to join and what makes them worthy of continued participation as well as why they stop participating.

Procedures

The online format was selected for a number of strategic reasons. First, it offers the best opportunity to take the study to a national scale. Second, it was clear to researchers that they would need to survey a large number of youth. They expected a significant diversity of responses as to why youth participate in after school programs and their patterns of participation; they hoped that the online format would facilitate reaching and gaining the interest of large numbers of youth. Third, the format offered the opportunity to target subpopulations of youth from across the United States in both urban and rural areas.

In our previous research, we have found that some subpopulations of youth are often underserved and that the programs that are offered may be poorly attended if they do not match the needs or interests of the young people they hope to serve (Borden, Perkins, Villarruel, & Stone, 2005; Villarruel, Perkins, Borden, & Keith, 2003). While sacrificing the benefits of a representative sample, the use of online methods provides a unique opportunity to include a large and diverse group of youth, which is essential to the success of the study.

The website that houses the survey was designed to be easily accessible and appealing to young people. The website’s designer, who has expertise in user-friendly survey technology, employed exciting and colorful design elements and made use of formatting familiar to young people to ensure that they would find the survey engaging enough to complete. The study is inclusive, as respondents don’t have to belong to particular organizations in order to participate. It is also completely anonymous—there is no identifying information requested in the survey. In addition, the data are secured—entered into a password protected database—and no one but the researchers can access them. All of these factors have contributed to the good reception of the study. In fact, a sizable number of youth have asked if they can fill out the survey again because they thought it was a positive experience.

Discussion

Computer-based evaluation systems offer researchers and practitioners the opportunity to provide computer access or to use paper and pencil surveys followed by data entry. This flexibility allows programs to determine which methodology would work best for their programs. The use of online printable surveys allows for quick and easy access to the information required for paper and pencil survey administration.

Online surveys provide the opportunity to develop survey techniques that can be used by younger children and non-readers because this technology can provide pictures and can even “read” the survey for the participant. Providing both audio and visual aids assists those taking the survey and thus expands the possibilities for a more diverse sample.

Another advantage of online data entry is that a system can also be set up to provide basic statistical analysis to the person who just entered the data. The analysis conducted can offer several options, such as demographics and frequencies. For pre- and post-test data, matched pair T-tests can be requested, with text-based results and interpretations returned to the practitioner. Thus, online data collection can offer both a system that contains validated assessment tools presented in an appealing computer-based format and illustrative and audio assessment appropriate for younger youth or less skilled readers.

Lessons Learned

One lesson learned in developing this online study was that the effort to make the protocol simple for participants did not make the process any easier for the evaluators. Dealing with the technology for creating and implementing the online survey proved to be very complex and challenging. Another hurdle associated with this new data collection technique was meeting a university institutional review board’s strict but reasonable criteria of anonymity and security to protect youth’s rights. This survey will be conducted for several years in order to allow as many young people as possible to participate.

References

Borden, L.M., Perkins, D. F., Villarruel, F.A., & Stone, M. (2005). To participate or not to participate: That is the question. In H. Weiss, P. Little, and S. Bouffard (Eds.) Participation In Youth Programs: Enrollment, Attendance, And Engagement. New Directions For Youth Development. 105: 33-49.

Carnegie Corporation (1992). A Matter of Time: Risk and Opportunity in the Out-of-School Hours. New York: Carnegie Corporation.

Eccles, J. S., & Gootman, J. A. (Eds.). (2002). Community programs to promote youth development. Washington, DC: National Academy Press.

Larson, R.W. (2000). Toward a psychology of positive youth development. American Psychologist, 55, 170-183.

Shields, C.M. (2003). Giving Voice to Students: Using the Internet for Data Collection. Qualitative Research, 3(3), 397-414.

Villarruel, F. A., Perkins, D. F., Borden, L. M., & Keith, J. G. (2003). Community youth development: Programs, polices and practices. Thousand Oaks, CA: Sage Publications.

William T. Grant Commission on Work, Family, and Citizenship (2000). The forgotten half: Pathways to success for America’s youth and young families. Philadelphia, PA: Public/Private Ventures.