Sage Journals: Discover world-class research

Abstract

The Delphi Technique has been useful in educational settings in forming guidelines, standards, and in predicting trends. Judd lists these major uses of the Delphi Technique in higher education: (a) cost-effectiveness, (b) cost–benefit analysis, (c) curriculum and campus planning, and (d) university-wide educational goals and objectives. The thorough Delphi researcher seeks to reconcile the Delphi consensus with current literature, institutional research, and the campus environment. This triangle forms a sound base for responsible research practice. This book gives an overview of the Delphi Technique and the primary uses of this technique in research. This article on the Delphi Technique will give the researcher an invaluable resource for learning about the Delphi Technique and for applying this method in educational research projects.

Keywords

educational research education social sciences Delphi Technique Delphi method guidelines consensus trends planning groupthink expert opinion expert consensus

No man alive, however young and strong, with mortal force alone could hope to budge that bed . . . for it contains a secret in its making. Within our court a long-leaved olive tree stood stout and vigorous, just like a pillar. Around that trunk I built our bridal room. I finished it with close-set stones and laid a roof above; I added doors that fitted faultlessly. Then I lopped off the olive’s long-leaved limbs; and so I thinned the trunk. I smoothed it down with craft and care; I made that wood true and straight, and it became my bedpost. Once I’d bored it with an auger, I, starting with that part began to shape my frame and, with that job well done, inlaid my work with silver, ivory, and gold. Inside the frame I stretched taut ox hide thongs; their crimson shone. My secret sign is told. Woman, I do not know if my bed stands where it once stood or if by now some man has sawed the bedstead from the trunk and set my bed elsewhere.

The Odyssey, Book XXIII

Definition

The Delphi Technique is a communication structure aimed at producing a detailed critical examination and discussion. Delphi studies have been useful in educational settings in forming guidelines, standards, and in predicting trends.

Background

The most notable use of the Delphi Technique was the RAND Corporation study conducted by Norman Dalkey and Olaf Helmer in 1963 to assess the direction of scientific breakthroughs, population control, automation, space progress, war prevention, and weapons systems (Jones, 1980). Many similar studies were patterned after the RAND study and used by the Department of Defense and other government agencies, factories, businesses, and health care agencies for the purpose of forecasting future trends and as a planning tool.

Researchers at the RAND Corporation in the 1960s jokingly referred to the research of Norman Dalkey and Olaf Helmer as Delphi research (Turoff & Hiltz, 1996). They applied this label because of the anonymous manner in which Dalkey and Helmer contacted nuclear science experts to gain information about future nuclear science trends and the forecasts that resulted from these queries.

The Delphic aura created by the anonymity and distance of the panelists continues as both a blessing and a curse for researchers. Proponents of the Delphi Technique agree that researchers can obtain more accurate data using questionnaires distributed to a group of anonymous experts at a distance than in face-to-face committee meetings where certain individuals tend to dominate the decision-making process (Delbecq, Van de Ven, & Gustafson, 1975; Linstone & Turoff, 1975; Moore, 1987). Critics of the Delphi Technique insist that the absence of social-emotional support makes the process too mechanical, non-motivating, and more disliked than liked among the respondents (Van de Ven, 1974).

The Delphi Technique is laden with many misconceptions as noted by Turoff and Hiltz (1996). Some of the following misconceptions noted by Turoff and Hiltz were as follows: (a) It is a method for predicting future events and for generating a quick consensus by a group; (b) it is the use of an anonymous survey to collect information; (c) it is a method for quantifying group judgment. Although these statements are partially true, they are often oversimplified or taken out of context by Delphi researchers. For example, Turoff and Hiltz concluded that reaching consensus is actually contrary to the purpose of a Delphi.

Delphi Technique in Educational Settings

The Delphi Technique had its beginnings in academia in the 1930s. Cyphert used the Delphi Technique at Ohio State University to develop criteria for evaluating the faculty at Ohio State (Fortune, personal communication, June 2, 1999). The Delphi Technique has not been widely used by educators until recently because it is both time-consuming and labor-intensive (Weingand, 1998). Electronic mail and computer analysis software make the Delphi Technique more appealing to modern researchers.

Delphi studies have been useful in educational settings in forming guidelines, standards, and in predicting trends. Judd (1972) lists five major uses of Delphi Techniques in higher education: (a) cost-effectiveness, (b) cost–benefit analysis, (c) curriculum and campus planning, (d) university-wide educational goals and objectives, and (e) generalized futuristic educational goals and objectives.

Lewis (1984) found that most of the Delphi studies in higher education were used to solve problems. In commenting on the likelihood that a Delphi study will have a strong impact in higher education, Lewis (1984) stated the following conditions: a solution to a recognized problem is actively being sought; the persons who will be affected and whose cooperation is needed are involved with the Delphi study; and the persons who conduct the Delphi are able to act upon the results.

The Delphi technique works well as an initial step in defining planning and marketing issues. Marketing has become a critical issue as more and more educational institutions compete to obtain necessary funding and support from academic and community sources.

The Delphi Technique will be useful for educators in developing curricula and learning experiences to prepare our students for future careers. These studies will be useful when coordinated with other grounded research in determining curricular needs, training and staffing needs, and for recruitment purposes.

Trend studies will be useful because they can be conducted representing a broad international base and the results can be easily disseminated via professional websites. Organizations using the Delphi technique in a setting where it will be broadly distributed should take care to explain this method and to encourage libraries to review the findings in terms of their own campus and benchmark findings.

Studies to determine issues and to establish guidelines are useful because they can involve an entire panel of experts on these issues. These experts volunteer to spend the minimal amount of time that is required to complete a Delphi questionnaire. The information that is collected from this questionnaire may be useful to more than one university and generally may be shared in aggregate form. Most universities could not afford to pay travel expenses and honoraria for all of these experts to come to one campus.

Delphi studies are extremely useful for collecting data from students and alumni regarding the curriculum, and information science trends, and funding. Some of these individuals may be hesitant to speak out in a focus group or other traditional forums. The Delphi Technique may also be used to further clarify or validate findings from surveys, focus groups, and interviews.

Expert Opinion

Helmer (1966) suggested applying the Delphi whenever policies and plans have to be based on informed judgment, and to some extent to any decision-making process. When formulating policies and plans based on expert opinion, it is important to recognize the fallacies inherent in human judgment.

Despite the fallacies inherent in seeking advice from the experts, the Delphi Technique can provide important insights from a panel of expert educators when developing standards, guidelines, and determining future trends.

Guidelines for Using the Delphi Technique

Linstone and Turoff (1975) have identified criteria that can be used to determine when the Delphi Technique should be used.

A problem does not lend itself to precise analytical techniques but can benefit from subjective collective judgments.

The individuals needed to contribute to the examination of a broad or complex problem have no history of adequate communication and may represent diverse backgrounds with respect to expertise or experience.

The heterogeneity of the participants must be preserved to assure validity of the results.

Purpose and Objectives

Socrates described himself as a gadfly “appointed to this city as though it were a large horse which because of its great size is inclined to be lazy and needs the stimulation of a stinging fly” (Plato, 1957, p. 17). The Delphi serves a similar purpose in education. Educators can uncover and discuss issues and problems in an anonymous forum that would not otherwise be brought to the forefront.

Developmental research

The Delphi Technique is useful for exploratory research and planning as indicated by Linstone and Turoff (1975), Moore (1987), and Delbecq et al. (1975). The Delphi Technique provides a useful means for exploring and describing current issues and problems and is useful for developmental research.

Isaac and Michael (1997) indicate that the purpose of developmental research is to ask questions about the patterns, direction, and sequence of growth or change and to explore the interrelated factors affecting these characteristics.

Research Procedures

Turoff and Hiltz (1996) state that the heart of the Delphi is the structure that relates all the contributions made by the panelists and produces a group view.

Turoff and Hiltz (1996) provide a detailed explanation of the Delphi phases as follows:

Formulation of the issues: What is the issue that really should be under consideration? How should it be stated?

Exposing the options: Given the issues, what are the policy options?

Determining initial positions on the issues: Which are the issues everyone agrees on and which are the unimportant ones to be discarded: Which are the ones exhibiting disagreement?

Exploring and obtaining the reasons for disagreements: What individual underlying assumptions, views, or facts are being used to support the panelists’ respective positions?

Evaluating the underlying reasons: How does the group view the separate arguments used to defend various positions, and how do they compare to one another on a relative basis?

Reevaluating: Reevaluation is based on the views of the underlying evidence and the assessment of its relevance to each position. (p. 88)

Panel Selection

Patton (1990) recommends including key experts in a subject field to solicit the latest thinking and to inform policy making. He recommends synthesizing expert opinion with existing opinion to pull together a research base for policy making. Andranovich (1995) recommends that the moderator establish a predetermined set of panelist qualifications and the number of panelists.

The Delphic probe

Andranovich (1995) suggests that the purpose of the Delphi must be clear so that the initial question can be developed. The Delphi question must elicit the information that is desired from the panelists. Delbecq et al. (1975) suggest focusing the Delphi question using the following three probes. Why are you interested in this particular Delphi? What do you need to know that you do not know now? How will results from the Delphi influence decision making once the procedure is completed?

The first Delphi probe might mention the need for providing individuals with disabilities equal access to academic library services and ask the panel to identify exhaustively the substantive issues and critical problems with using assistive technologies in academic libraries and to suggest corresponding guidelines for addressing these issues and problems (Green, 1999). The results may be used along with other research to provide a basis for establishing standards and evaluating assistive technologies services.

The Delphi Process

Stewart and Shamdasami (1980) outline the steps in the Delphi process as follows:

Develop the initial Delphi probe or question;

Select the expert panel;

Distribute the first round questionnaire;

Collect and analyze Round 1 responses;

Provide feedback from Round 1 responses, formulate the second questionnaire based on Round 1 responses and distribute;

Repeat Steps 4 and 5 to form the questionnaire for Round 3;

Analyze final results;

Distribute results to panelists.

Typically, Delphi studies include three rounds. The purpose of the first round is to form issues. The second round provides the panelists with feedback from the first round and presents a questionnaire to the panelists. The panelists rate the items on the questionnaire using a predetermined scale. The Delphi moderator uses measures of central tendency to determine consensus from the second round. Individuals are asked to reevaluate their opinions in the third round when they differ significantly from the other panelists. The purpose of the third round is to provide feedback from the previous round and to reach a final consensus or to indicate that consensus cannot be reached.

Measures of central tendency are used to present and determine the consensus. Individuals who express different views from other panelists may be asked to provide reasons for their dissenting views to clarify their positions.

Comparisons are made between the panelists’ views and the literature. The purpose of this comparison is to triangulate expert opinion with the literature. Lincoln and Guba (1985) emphasize the importance of triangulating qualitative research using the constructs or canons of credibility, transferability, dependability, and confirmability.

Scale Development

Cyphert and Gant (1970) describe the advantage of using the modal score as being that “prospective participants must be made to feel that their response is valid so that they will take part” (p. 273). They insist as does Stag (1983) that using the mode rather than the interquartile range gives the panel member a greater affiliation with the study. Showing a mean score does not give an accurate view of the individual panelists’ ratings.

Using the modal score may in some cases yield bimodal or trimodal responses. The indication is that consensus could not be achieved on items that continue to have bimodal scores in the third round (Stag, 1983).

Cox (1996) recommends using a scale with an even number of points so that the respondent cannot circle a number in the middle and is forced to choose one side or the other. This is particularly important in designing a scale for a Delphi study because mid-range responses can lead to a false consensus. Even numbered scales can be more easily collapsed into fewer categories (Cox, 1996). Spector (1992) states that to develop a scale, a definition must be adopted. A list of scale definitions should be provided for the study.

Data processing and analysis

Palomba and Banta (1999) insist that assessment is more than a collection of data. They emphasize the importance of using assessment results to improve educational programs. Wainwright and Dean (1976) assert, “no single technique may be regarded as adequate for complete evaluation purposes, nor does the whole range of techniques available provide any certainty of arriving at measures of collection adequacy which may be irrefutable” (p. 82). Isaac and Michael (1997) suggest that a study must be confirmed by two or more independent measurement processes to reduce the uncertainty of its interpretation. Webb, Campbell, Schwartz, and Sechrest (1966) state that “the triangulation process is far more powerful evidence supporting a proposition than any single approach” (p. 93). Morgan (1995) also recommends that a wide variety of techniques should be deployed including questionnaires, checklists, interviews, and simulations in the data processing and assessment process to give a rich picture approach. Rossman and Wilson (1985) discuss the richness that is brought to research when data from different sources are used to corroborate, elaborate, or illuminate the research (p. 144).

Palomba and Banta (1999) state, “of all the important factors in creating a successful assessment program, none matters more than widespread involvement of those who are affected by it” (p. 53). Creswell (1998) agrees that verification especially from different frames of reference helps to underscore the legitimacy of results in qualitative research. Jones (1980) notes that experts often do not have time to pursue issues outside of their disciplines. For this reason, it is difficult to form a holistic picture using the tunnel vision of a pool of experts. Linstone and Turoff (1975) stress that the Delphi does not substitute for staff studies, committee deliberations, or other decision-making forums. “Rather, it organizes and clarifies views in an anonymous way, thereby facilitating and complementing the committee’s work” (p. 75).

Morgan (1995) recommends a participatory management approach when implementing new goals to build a relationship of trust with employees. Abbott (1994) suggests that the planning should not be isolated from the normal management process, nor should one person develop it. Seelman (1999) stresses the importance of making use of all available data in the planning process.

Elements for Interpreting the Delphi in an Educational Setting

The following elements should be used when applying the guidelines suggested by the Delphi Study in an academic setting. These items are included in the Expert Consensus in Academia Model included at the end of this article (see Figure 1).

Figure 1.

Expert consensus in academia model.

Campus environment. Cooperative efforts with the faculty, staff, administrators, instructional design specialists, and students are essential when planning and implementing guidelines and services. Cooperative efforts assure maximization of funding and greater service efficiency.

Consensus. Consensus is reached after several rounds in a Delphi study using measures of central tendency. The mode gives the most accurate picture of the views that have been expressed by the experts. The mean can give a false consensus because it takes in the views expressed by the extreme outliers. Delphi consensus should be weighed against existing campus policy, instructional design principles, and the campus environment.

Focus groups or survey data. User services data will provide another useful check for Delphi findings.

Institutional research. Institutional research provides another check for verifying Delphi findings. Campus demographics, statistics, user surveys, and other institutional reports should be considered when interpreting the findings of the Delphi study.

Instructional design principles. Guidelines that are carefully constructed using accepted instructional design principles help to confirm a seamless transition within an educational setting.

Prior research. The Delphi findings should be checked for congruency with prior research. The researcher should be prepared to explain discrepancies between the Delphi study and scholarly literature.

Limitations

Armstrong (1985) states that evidence in favor of experts in Delphi studies is lacking. Jones (1980) insisted that a theory should be explicit and consistent, and that a Delphi study is constructed by piecing together a sometimes incompatible set of opinions. Sackman (1976) refers to studies in which similar results were obtained with students that had been obtained from the experts. It is possible that the panel of experts may not reach a consensus on some issues, thus providing only fragmented information (Combs, 1985). Lawrence (1980) noted that developmental research may not always be reliable because of poor sampling procedures or an invalid instrument and could be detrimental in the decision-making process when used in isolation from other methods.

Gordon and Helmer (1966) acknowledged some of the following weaknesses in the Delphi Technique: Instability of panel membership (high drop-out rates) and respondents’ competence to answer questions outside their specialized fields.

Fink, Kosecoff, Chassin, and Brook (1984) noted that the Delphi Technique sometimes produces the lowest common denominator of agreement. Rennie (1981), likewise, lamented that the Delphi provides bland generalities representing the lowest common denominator of debate.

Anonymity is one of the characteristics of the Delphi Technique. This characteristic can detract from the credibility of the study and can make the experts inaccessible to future researchers and practitioners.

Linstone (1975) cautioned against the following pitfalls involved with using and interpreting the Delphi Technique that are applicable to this study:

The simplification urge. Simplistic misjudgments can easily result from the basic statements that are inferred from a Delphi study.

Illusory expertise. The specialist is not necessarily the most knowledgeable person. Sometimes experts lack the ability to see the global picture thus, thwarting their ability to produce effective organizational decisions.

Sloppy execution. Superficial analysis of responses is a most common weakness. Hasty or incomplete answers can give the moderator an inadequate or inaccurate picture.

Overselling. Linstone cautions against the pitfalls of inbreeding (repeating Delphi studies on the same subject, using the same experts and anonymity). Anonymity may be a disadvantage in that the source of a statement may be far more significant than its substance. “Consensus of several participants may be of less value than knowledge of their identity” (Linstone, 1975, p. 585).

Deception. The Delphi process is not immune to manipulation or propaganda use. The anonymity in such a situation may even facilitate the deception process (Linstone, 1975, p. 586). Welty (1971) uses the analogy of the Greek myth of Ino, the wife of King Athamus of Orchomenus. When the King dispatched a messenger to the Oracle of Delphi, Ino bribed him to return with a falsified story. In a second consultation at Delphi, the Oracle based its pronouncements on the false version of the first utterances.

Results

Gordon and Helmer (1966) suggest that the facilitator should ensure the following circumstances to derive the most reliable results from the Delphi: (a) the panel membership should remain reasonably stable; (b) time lapses between questionnaires should be held to a minimum; (c) questions should be unambiguous; and (d) feedback should be provided that gives reasons for consensus opinions and consensus using the mean or average values should be avoided to avoid discriminating against outliers.

Zargari, Campbell, and Savage (1999); Polanin (1990); and Scheele (1975) all insist on the importance of including the stakeholders in the Delphi study to insure that the study will be implemented or will have the desired impact. The stakeholders should recognize the relevance of the problem and would be equipped to act on the results.

Summary

The Delphi Technique is a communication structure aimed at producing a detailed critical examination and discussion. The Delphi Technique is an iteration of anonymous questionnaire responses to achieve consensus by an expert panel (Stag, 1983). Lincoln and Guba (1985) emphasize the importance of triangulating qualitative research using the constructs or canons of credibility, transferability, dependability, and confirmability. The Expert Consensus in Academia Model at the end of this article reflects the importance of previous research in interpreting a Delphi study and views the findings of the Delphi in light of instructional design principles, campus environment, and respects the views of non-experts who will be involved in the decision-making process.

When the great Odysseus returned home, the ever-faithful Penelope wanted to prove that he was in fact her long lost husband and not an imposter. As a final test, she instructed the servants to move the bed that he had made out in to the hall for him. Odysseus became enraged at the mention of moving the bed because he knew that the bed could not be moved unless someone had sawed down the olive tree around which it had been built.

Odysseus had constructed the bed around a live olive tree that was on the building site before he built the house. He framed the bedchamber around the bed and built the house around the bedchamber. Thus, the live olive tree was the foundation for the house. By extension, the framing of the bed and Odysseus and Penelope’s relationship were symbolic of the stability of Greek society. Homer’s epic implied to Greek citizens that they were a part of a city-state that had been framed as carefully as Odysseus’ bed. This epic hero provided the Greeks with the hope that their future would be as steadfast and unmovable as the great Odysseus’ bed (Vandiver, 1999).

The framing of research within the modern context involves the triangulation of many elements. The thorough researcher must carefully construct a project that is as solid as Odysseus’s bed. Delphi studies that incorporate mixed methodologies to corroborate the opinion of experts can be most useful in an educational setting.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Funding

The author(s) received no financial support for the research and/or authorship of this article.

Author Biography

Ravonne A. Green has taught educational research classes and library and information science courses. She has also written a textbook about program evaluation titled Case Study Research: A Program Evaluation Guide for Librarians .

References

Abbott

(1994). Performance measurement in library and information services. London, England: Aslib.

Andranovich

(1995). Developing community participation and consensus: The Delphi Technique. Los Angeles: California State University.

Armstrong

(1985). Long-range forecasting: From crystal ball to computer (2nd ed.). New York, NY: Wiley.

Combs

(1985). Education in the year 2035—A Delphi study to identify possible futures of the public secondary school (Doctoral dissertation). Virginia Polytechnic Institute and State University, Blacksburg.

Cox

(1996). Your opinion please: How to build the best questionnaires in the field of education. Thousand Oaks, CA: Corwin Press.

Creswell

(1998). Qualitative inquiry and research design: Choosing among five traditions. Newbury Park, CA: Sage.

Cyphert

Gant

(1970). The Delphi Technique: A tool for collecting opinions in teacher education. Journal of Teacher Education, 21, 417-425.

Delbecq

Van de Ven

Gustafson

(1975). Group techniques for planning: A guide to nominal group and Delphi processes. Glenview, IL: Scott, Foresman.

Fink

Kosecoff

Chassin

Brook

(1984). Consensus methods: Characteristics and guidelines for use. American Journal of Public Health, 74, 979-983.

10.

Gordon

Helmer

(1966). Report on a long-range forecasting study. In Gordon

Helmer

(Eds.), Social technology (pp. 7-9). New York, NY: Basic Books.

11.

Green

(1999). Assistive technologies for individuals with print disabilities in academic libraries (Doctoral dissertation). Virginia Polytechnic Institute and State University, Blacksburg.

12.

Helmer

(1966). The use of the Delphi Technique in problems of educational innovations (P-3499). Santa Monica, CA: The RAND Corporation.

13.

Isaac

Michael

(1997). Handbook in research and evaluation: A collection of principles, methods, and strategies useful in the planning, design, and evaluation of studies in education (4th ed.). New York, NY: Edits Publications.

14.

Jones

(1980). Options for the future: A comparative analysis of policy-oriented forecasts. New York, NY: Praeger.

15.

Judd

(1972). Forecasting to consensus gathering: Delphi grows up to college needs. College & University Business, 53(1), 35-38, 43.

16.

Lawrence

(1980). Application of the Delphi Technique in determining automotive technologist curriculum content (Doctoral dissertation). Virginia Polytechnic Institute and State University, Blacksburg.

17.

Lewis

(1984). Characteristics of selected Delphi studies and their perceived impact in higher education (Doctoral dissertation, University of Florida, Order No. DA842938). Ann Arbor, MI: University Microfilms International.

18.

Lincoln

Guba

(1985). Naturalistic inquiry. Newbury Park, CA: Sage.

19.

Linstone

H. A.

(1975). Eight basic pitfalls: A checklist. In Linstone

H. A.

Turoff

(Eds.), The Delphi method: Techniques and applications (p. 418). London, England: Addison-Wesley.

20.

Linstone

H. A.

Turoff

(1975). General applications: Policy Delphi. In Linstone

H. A.

Turoff

(Eds.), The Delphi method: Techniques and applications (pp. 311-329). London, England: Addison-Wesley.

21.

Moore

(1987). Group techniques for idea building, Vol. 9: Applied social research methods series. Newbury Park, CA: Sage.

22.

Morgan

(1995). Performance assessment in academic libraries. New York, NY: Mansell.

23.

Palomba

Banta

(1999). Access essentials: Planning, implementing, & improving assessment in higher education. San Francisco, CA: Jossey-Bass.

24.

Patton

(1990). Qualitative evaluation and research methods (2nd ed.). Newbury, CA: Sage.

25.

Plato. (1957). Laches. In Bowra

(Ed.), The Greek experience. New York, NY: Barnes & Noble.

26.

Polanin

(1990). Technical core competencies for computer integrated manufacturing technicians (Doctoral dissertation). University of Illinois at Urbana–Champaign.

27.

Rennie

(1981). Consensus statements. New England Journal of Medicine, 304, 665-666.

28.

Rossman

Wilson

(1985). Numbers and words: Combining quantitative and qualitative methods in a single large-scale evaluation study. Quality & Quantity, 28, 315-327.

29.

Sackman

(1976). Toward more effective use of expert opinion: Preliminary investigation of participatory polling for long-range planning (P-5570). Santa Monica, CA: The RAND Corporation.

30.

Scheele

(1975). Reality construction as a product of Delphi interaction. In Linstone

H. A.

Turoff

(Eds.), The Delphi method: Techniques and applications. Boston, MA: Addison-Wesley.

31.

Seelman

(1999, February 1). Tech Act project directors meeting. Retrieved from http://www.capcsd.org/proceedings/2002/talks/kseelman2002pdf

32.

Spector

(1992). Summated rating scale construction: An introduction, Vol. 82: Applied social research methods series. Newbury Park, CA: Sage.

33.

Stag

(1983). A study of an Arkansas community college general education and business core curriculum using the Delphi (Doctoral dissertation). The University of South Dakota, Vermillion.

34.

Stewart

Shamdasani

(1980). Focus groups: Theory & practice, Vol. 20: Applied social research methods series. Newbury Park, CA: Sage.

35.

Turoff

Hiltz

(1996). Computer based Delphi processes. In Adler

Ziglio

(Eds.), Gazing into the oracle: The Delphi Technique and its application to social policy and public health. London, England: Kingsley.

36.

Van de Ven

(1974). Group decision-making and effectiveness. Organization and the Social Sciences, 5(3).

37.

Vandiver

(1999). The Odyssey of Homer. Springfield, VA: The Teaching Company.

38.

Wainwright

Dean

(1976). Measures of adequacy for library collections in Australian colleges of advanced education: Report of a research project conducted on behalf of the Commission on Advanced Education (Vol. 2). Perth: Western Australia Institute of Technology.

39.

Webb

Campbell

Schwartz

Sechrest

(1966). Unobtrusive measures: Nonreactive research in the social sciences. Chicago, IL: Rand McNally.

40.

Weingand

(1998). Future-driven library marketing. Chicago, IL: ALA.

41.

Welty

(1971). A critique of some long-range forecasting developments. Bulletin of the International Statistical Institute, 44, 403-408.

42.

Zargari

Campbell

Savage

(1999). Determination of curriculum content and requirements for a Doctor of Philosophy degree program in industrial technology. Retrieved from http://borg.lib.vt.edu/ejournals/JITE/v32n4/zargari.html