You can also download the pdf file by click here.

Analytical Framework

The Literature Search and Selection

Studies included in the research synthesis were identified through a three-step process. First, we conducted a thorough search for all studies included in ERIC (1966-2002) through FirstSearch with the following keywords: (distan* and education, distan* and learning, distan* and teaching, distan* and instruction, online and education, online and learning, online and teaching, online and instruction, on-line and education, on-line and learning, on-line and teaching, on-line and instruction, web-based and education, web-basedand learning, web-based and teaching, web-based and instruction, virtual and education, virtual and learning, virtual and teaching, virtual and instruction). The search identified 8840 potentially relevant articles. Citation information for all 8840 articles was then transferred into EndNote (5.0) to build the first database.

At the second stage, the database was further examined based on the following criteria:

1. The article had to be published in journals. The decision of including only journal articles was based on the concern of study quality. Previous research reviews on distance education had pointed out the low quality problem of most studies. We believed that journal articles were of higher quality because of peer review procedures. Only including journal articles may result in publication bias but we believed that the risk was minimal as there had not been a dominant paradigm for distance education over the years to cause a certain bias against or for positive, negative, or non-significant findings.

2. The article must have complete reference information (author, date, source, etc.)

3. The article had to include at least one evaluation study of distance education. The specific outcome measured was not limited.

4. The article must have at least one comparison study on distance education and face-to-face education. Studies in which students’ own pre-treatment scores served as controls for their post-treatment scores and those in which one distance course was compared with another distance course were excluded.

5. The article must have some empirical data. Articles were not included if they merely describe a distance education course.

6. The article had to include enough statistical information for computing an effect size. The specific information we were looking for was means, standard deviation, sample size for both the distance education group and the face-to-face group, or t value, F value and degree of freedom (df).

At this stage, 1100 articles that either were not journal articles or didn’t have complete reference information were removed from the database. Then the research team read the abstracts of all 7740 articles, of which, 6365 articles were removed because they didn’t meet criterion 3 or 4. Articles that could not be clearly decided at this stage were kept in the database to be dealt with at a later stage. A total of 1375 references were left after this selection.

At the third stage, the research team collected and read all the 1375 articles and excluded those that didn’t have empirical data. As a result, 421 articles were left after this elimination. The research team then read and coded all 421 articles and found that only 49 articles contained sufficient information for calculating the effect size. For fear of missing some articles that may actually have the information due to the large number of articles and complexity of the database, the research team examined all 421 articles one more time and identified 2 more articles that have complete information to calculate the effect size. Ancestry search was conducted but no extra articles met all the criteria were found. Finally, 51 journal articles were included for the final analysis.

Analytical Framework

To identify those methodological and substantive characteristics that may be responsible for significant variations in the findings, a detailed analytical framework was developed through an iterative process. The framework was developed based on our understanding of possible sources of variation in the studies. The heterogeneity in outcomes across studies can come from three sources: the publication, the study, and the instruction. Figure 1 depicts the logic model underlying the analytical framework.

Figure 1: Logic Model of Distance Education Effectiveness

For publication and study features, we started with Stock (1994)’s seven categories for describing research reports: report identification, the setting of the study, subjects, methodology, treatment characteristics, statistical outcomes or effect sizes, and coding process. Instructional features refer to the characteristics of the distance education program under study. It has been argued that distance education should be considered as education at a distance (Shale, 1990):

 In sum, distance education ought to be regarded as

 education at a distance. All of what constitutes the

 process of education when teacher and student are able

 to meet face-to-face also constitutes the process of

 education when teacher and student are physically

 separated. (p. 334)

In other words, the quality of distance education programs is influenced by the same set of factors that affect the quality of face-to-face education. Schwab(Schwab, 1983) characterizes education in terms of four common places of education: teacher, student, what is taught, and milieux of teaching-learning. This characterization is also applicable to the study of distance education. While teacher, student, and what is taught remain pretty much the same as face-to-face education, the milieux of teaching-learning is different between distance and face-to-face education in that the milieux of teaching-learning of distance education is mostly mediated through some kind of technology. Hence we describe the milieux of teaching-learning in terms of the format and method of delivery. Finally, we used the grounded theory(Glaser, 1992; Glaser & Strauss, 1967) to guide the development of the framework. Following a typical constant-comparison process, we started coding the studies with the initial framework and then modified the framework when we came across new features during the coding process. Table 1 summarizes the variables included in the final framework. In the following paragraphs, we describe these variables and reasons for their inclusion in the framework.

Table 1. Variables Analyzed in this Study

Category	Variable	Scale	Source
Publication Features	Publication Year		Publication
	Instructor as Author	Yes/No	Publication
Study Features	Study Design	Quasi-experimental or Experimental	Researcher Coding
	Measurement	standardized test, Researcher-developed instruments, Instructor-developed instruments, and Published instrument	Publication
	Indicators of effectiveness	Grades, student satisfaction, faculty satisfaction, standardized tests	Publication
	Results	Sample size, mean, standard deviation, t value, p value	Publication
Instructional Features: Instructor	Instructor Involvement	1 to 10 1=No involvement 10=Complete involvement	Researcher Coding
	Status	Professor, graduate student or professional	Publication
Instructional Features: Learner	Background	High school diploma, or college degree	Publication
	Status	Full time or part time	Publication
Instructional Features: Curriculum	Content Area	Subject taught	Publication
	Degree	Yes/no	Publication
	Credit	Yes/no	Publication
	Course Time		Publication
	Class Time		Publication
Instructional Features: Milieux	Interaction type	Asynchronous interaction, or Synchronous interaction, or both, or Non-interactive involvement	Publication
	Media involvement	1 to 10 1=No involvement 10=Complete involvement	Researcher Coding
	Setting	K-12,Graduate, Undergraduate, Military, etc.	Publication

Evidence of Effectiveness

There are different ways to measure the effectiveness of distance education programs. Studies in distance education thus differ in what they used as evidence of effectiveness and the reliability and validity of the evidence used. The variation in what was measured and the quality of the measurement may explain the heterogeneity of outcomes.

Outcome measures. Information about what has been used to assess the effectiveness of distance programs was collected for each study. A study could use the one or more of following measures: Grades, Quizzes, Independent/standardized Tests, Student Satisfaction, Faculty Satisfaction, Dropout Rate, Student Evaluation of Learning, Student Evaluation of Course, External Evaluation and Cost Effectiveness. Grades usually are the final scores students received for the class. Student Evaluation of Learning is students’ perception of how much they learned from this course, which can be significantly different from the grades they received. Cost Effectiveness is the ratio of costs and students achievements.

Source of Instrument. The source of instruments used to measure effectiveness can affect the final outcomes in that instruments from different sources may have different levels of reliability and validity. We identified four sources of the most frequently used measures: commercial testing agencies, the researcher (the author of the article), the instructor of the course, and publishers of textbooks that include assessment items.

Study design. The design of a study is a good indication of its quality and thus the quality of the results. We were curious about whether certain type of design is associated with the study results, thus we coded the studies into two categories: true experimental or quasi-experimental. The differentiating characteristic between the two designs is whether random sampling method was used.

Study Results. To calculate effect sizes, the results of each study were recorded in the database. The study results include means, standard deviations, t values, F values, and r values, depending on what is reported in the primary study.

Factors Affecting Effectiveness

Factors that may affect the effectiveness of a distance education program can be categorized into two groups: publication features and instructional features. Two publication features were identified as possible factors affecting effectiveness: publication year and instructor as author.

Publication Year. Previous research found that the time when a study was conducted has a significant correlation with the reported effectiveness(Machtmes & Asher, 2000). The hypothesis was that technology used to deliver distance education has changed dramatically over years and that newer technologies seem to have more capacity to deliver richer and more powerful learning experiences. To verify this hypothesis, we recorded the year when a study was published for each article.

Instructor as Author. All studies are based on advocacy (Begg, 1994). We hypothesize that if the instructor of the distance learning course was also the author of the publication, the result could be more likely to favor distance learning. To verify this hypothesis, we coded whether the author of an article was also the instructor of the distance education program under study.

As mentioned before we used Schwab’s four common places of education to guide the identification of instructional features that can potentially affect the effectiveness of distance education programs: the teacher, the student, the curriculum, and the milieux. In each of the four common places are a number of potential factors that contribute the outcomes of learning.

Teacher: Instructor Involvement. The extent to which the instructor of a distance education course is involved in the actual delivery of the content and available for interactions with students during and outside the class sessions is termed ‘instructor involvement.” The level of instructor involvement is perhaps one of the most defining differences between traditional face-to-face education and distance education. In face-to-face education, the instructor generally delivers the content live and interacts with students both in and outside class meetings whereas in distance education programs the level of instructor involvement varies a great deal, from one extreme where the content is pre-programmed and delivered through some technology means without the actual involvement of an instructor to another where the instructor actually delivers the content live and is available for interactions with students in very much the same fashion as face-to-face education. Interactions between the teacher and students have been found to affect the quality of student experiences and learning outcomes in distance education (The Institute for Higher Education Policy, 2000). How content is delivered should also have an effect on student learning experiences and outcomes. This is also a hot topic for distance education programs because one of the appeals of distance education is the potential to increase efficiency by reducing the demand of actual involvement of faculty. Having one faculty to teach thousands of (or even more) students with the help of broadcasting, recording, and computing technologies has been a dream for many advocates of distance education. However, if higher-level of instructor involvement becomes a requirement for effective quality programs, the efficiency dream may never be realized. On the other hand, if the level of instructor involvement is found to be irrelevant to student outcomes, it would be unnecessary to assign an instructor to only a small group of students or be actually involved in the teaching. A pre-programmed video or computer program can accomplish as much. To assess the value of instructor involvement, we hence included this factor. We coded instructor involvement on a scale from 1 to 10, with 1 indicating no human instructor involvement (e.g., computer-based training) and 10 full involvement of a human instructor (e.g., two-way interactive TV courses).

Teacher: Status of the Instructor. Another way distance education programs have sought to increase efficiency is to employ non-regular faculty, who normally costs less than regular faculty. Thus we are interested in finding out whether instructor status influence student learning in distance education programs.

Teacher: Training for Teaching Distance Courses. It has been argued that instructors of distance education programs should be trained first because distance education is a different teaching environment from face-to-face classrooms. Again, the training has cost implications for programs. To test whether training affects student learning, we collected information about teacher training from each study, if that information is available.

Student: Education Level. We collected information about student’s educational attainment level before attending the distance course to examine whether, and if so which level, certain types of students are more prepared to take distance education courses.

What is Being Taught: Content Area. Some content may be more suited for distance education while others may be better taught in face-to-face course. Interested in finding if this assumption is true and if so, what content area is better suited for distance education, we collected information about the content area of each study. A course can be teaching one of the following subject areas: Social Science, Mathematics, Science, Medical Science, Literacy, Humanities, Business, Law, Engineering, Computer Science, Teacher Education, and Skills. Skills here represent any professional training that doesn’t fall into other categories. We coded medical sciences, business education, and teacher education separately because they have been one of the most commonly taught content areas in distance education.

The Milieux: Instructional Level. Distance education programs have been traditionally intended for adults but recently it has expanded to include younger audiences. As a related factor to student characteristics and content, the instructional level of distance education may be associated with its effectiveness. We grouped the distance education programs in each study into 10 levels: K-2(Lower elementary), 3-5(upper elementary), 6-9(middle school), 10-12(high school), Associate Degree(community college), Undergraduate Level(4 year college), Graduate Level, Professional Development, and Military Training. We also collected data about whether the course is for credit or not and whether it is for degree granting program or not.

The Milieux: Interaction Type. Interaction type characterizes how instructors and students interact in the distance learning process. There are four types of interaction: asynchronous, in which a time lag exists between the interactions of the instructor and students in that students may ask a question via email, to which the instructor may respond two days later; synchronous, where the potential exists for instructors and students to interact at the same time; non-interactive, where there is no interaction between instructors and students at all; and both synchronous and asynchronous interaction, where the instructor can interact with students both synchronously and asynchronously.

The Milieux: Media Involvement. Distance education programs also vary in the level of technology is used. Some programs employ a “mixed-model,” in which part of the instruction is conducted face-to-face whereas some others are completed delivered via technology. Proponents of the “mixed-model” suggest that some face-to-face contact is necessary or desirable to maintain student motivation and thus higher quality of education. We are interested in testing this hypothesis. Thus we coded each study’s level of media involvement, which is defined as the extent a certain instructional delivery system has been mediated by technologies, i.e., how frequently technology is used in a program. Media Involvement is coded on a scale from 1 to 10, where 1 indicating no technology was used while 10 indicating that the instruction was delivered completely with technology.