Likert Scale 1a2fk

Likert scale From Wikipedia, the free encyclopedia

Jump to: navigation, search A Likert scale (pronounced /ˈlɪkərt/,[1] also /ˈlaɪkərt/), or more accurately a Likert-type scale, is a psychometric scale commonly used in questionnaires, and is the most widely used scale in survey research, such that the term is often used interchangeably with rating scale even though the two are not synonymous. When responding to a Likert questionnaire item, respondents specify their level of agreement or disagreement on a symmetric agree-disagree scale for a series of statements. Thus the scale captures the intensity of their feelings.[2] The scale is named after its inventor, psychologist Rensis Likert.[3]

Contents [hide] •

1 Sample question presented using a five-point Likert item

•

2 Scoring and analysis

•

3 Level of measurement

•

4 Rasch model

•

5 Pronunciation

•

6 See also

•

7 References

•

8 External links

[edit] Sample question presented using a five-point Likert item An important distinction must be made between a Likert scale and a Likert item. The Likert scale is the sum of responses on several Likert items. Because Likert items are often accompanied by a visual analog scale (e.g., a horizontal line, on which a subject indicates his or her response by circling or checking tick-marks), the items are sometimes called scales themselves. This is the source of much confusion; it is better, therefore, to reserve the term Likert scale to apply to the summated scale, and Likert item to refer to an individual item. A Likert item is simply a statement which the respondent is asked to evaluate according to any kind of subjective or objective criteria; generally the level of agreement or disagreement is measured. It is considered symmetric or "balanced" because there are equal amounts of positive and negative positions.[4] Often five ordered response levels are used, although many psychometricians advocate using seven or nine levels; a recent empirical study[5] found that a 5or 7- point scale may produce slightly higher mean scores relative to the highest possible attainable score, compared to those produced from a 10-point scale, and this difference was statistically significant. In of the other data characteristics, there was very little difference among the scale formats in of variation about the mean, skewness or kurtosis. The format of a typical five-level Likert item is:

1. Strongly disagree 2. Disagree 3. Neither agree nor disagree 4. Agree 5. Strongly agree Likert scaling is a bipolar scaling method, measuring either positive or negative response to a statement. Sometimes a four-point scale is used; this is a forced choice method[citation needed] since the middle option of "Neither agree nor disagree" is not available. Likert scales may be subject to distortion from several causes. Respondents may avoid using extreme response categories (central tendency bias); agree with statements as presented (acquiescence bias); or try to portray themselves or their organization in a more favorable light (social desirability bias). Deg a scale with balanced keying (an equal number of positive and negative statements) can obviate the problem of acquiescence bias, since acquiescence on positively keyed items will balance acquiescence on negatively keyed items, but central tendency and social desirability are somewhat more problematic.

[edit] Scoring and analysis After the questionnaire is completed, each item may be analyzed separately or in some cases item responses may be summed to create a score for a group of items. Hence, Likert scales are often called summative scales. Whether individual Likert items can be considered as interval-level data, or whether they should be considered merely ordered-categorical data is the subject of disagreement. Many regard such items only as ordinal data, because, especially when using only five levels, one cannot assume that respondents perceive all pairs of adjacent levels as equidistant. On the other hand, often (as in the example above) the wording of response levels clearly implies a symmetry of response levels about a middle category; at the very least, such an item would fall between ordinal- and interval-level measurement; to treat it as merely ordinal would lose information. Further, if the item is accompanied by a visual analog scale, where equal spacing of response levels is clearly indicated, the argument for treating it as interval-level data is even stronger. When treated as ordinal data, Likert responses can be collated into bar charts, central tendency summarised by the median or the mode (but some would say not the mean), dispersion summarised by the range across quartiles (but some would say not the standard deviation), or analyzed using non-parametric tests, e.g. chi-square test, Mann–Whitney test, Wilcoxon signedrank test, or Kruskal–Wallis test.[6] Parametric analysis of ordinary averages of Likert scale data is also justifiable by the Central Limit Theorem, although some would disagree that ordinary averages should be used for Likert scale data. Responses to several Likert questions may be summed, providing that all questions use the same Likert scale and that the scale is a defendable approximation to an interval scale, in which case they may be treated as interval data measuring a latent variable. If the summed responses fulfill these assumptions, parametric statistical tests such as the analysis of variance can be applied. These can be applied only when more than 5 Likert questions are summed.[citation needed] Data from Likert scales are sometimes reduced to the nominal level by combining all agree and disagree responses into two categories of "accept" and "reject". The chi-square, Cochran Q, or McNemar test are common statistical procedures used after this transformation.

Consensus based assessment (CBA) can be used to create an objective standard for Likert scales in domains where no generally accepted standard or objective standard exists. Consensus based assessment (CBA) can be used to refine or even validate generally accepted standards.

[edit] Level of measurement The five response categories are often believed to represent an Interval level of measurement. But this can only be the case if the intervals between the scale points correspond to empirical observations in a metric sense. In fact, there may also appear phenomena which even question the ordinal scale level. For example, in a set of items A,B,C rated with a Likert scale circular relations like A>B, B>C and C>A can appear. This violates the axiom of transitivity for the ordinal scale.

[edit] Rasch model Likert scale data can, in principle, be used as a basis for obtaining interval level estimates on a continuum by applying the polytomous Rasch model, when data can be obtained that fit this model. In addition, the polytomous Rasch model permits testing of the hypothesis that the statements reflect increasing levels of an attitude or trait, as intended. For example, application of the model often indicates that the neutral category does not represent a level of attitude or trait between the disagree and agree categories. Again, not every set of Likert scaled items can be used for Rasch measurement. The data has to be thoroughly checked to fulfill the strict formal axioms of the model.

[edit] Pronunciation Rensis Likert, the developer of the scale, pronounced his name 'lick-urt' with a short "i" sound.[7] [8] It has been claimed that Likert's name "is among the most mispronounced in [the] field."[9] Although many people use the long "i" variant ('lie-kurt'), those who attempt to stay true to Dr. Likert's pronunciation use the short "i" pronunciation ('lick-urt').

[edit] See also •

Analog scale

•

Bogardus Social Distance Scale

•

Consensus based assessment (CBA)

•

Diamond of opposites

•

Discan scale

•

F-scale

•

Guttman scale

•

Ipsative

•

Mokken scale

•

Phrase completion scales

•

ProScan Survey

•

Rating scale

•

Rating sites

•

Reverse coding

•

Rosenberg self esteem scale

•

Satisficing

•

Semantic differential

•

Thurstone scale

•

Voting system

[edit] References 1. ^ Wuensch, Karl L. (October 4, 2005). "What is a Likert Scale? and How Do You Pronounce 'Likert?'". East Carolina University. http://core.ecu.edu/psyc/wuenschk/StatHelp/Likert.htm. Retrieved April 30, 2009. 2. ^ Burns, Alvin; Burns, Ronald (2008). Basic Marketing Research (Second ed.). New Jersey: Pearson Education. pp. 245. ISBN 978-0-13-205958-9. 3. ^ Likert, Rensis (1932). "A Technique for the Measurement of Attitudes". Archives of Psychology 140: 1–55. 4. ^ Burns, Alvin; Burns, Ronald (2008). Basic Marketing Research (Second ed.). New Jersey: Pearson Education. pp. 250. ISBN 978-0-13-205958-9. 5. ^ Dawes, John (2008). "Do Data Characteristics Change According to the number of scale points used? An experiment using 5-point, 7-point and 10-point scales". International Journal of Market Research 50 (1): 61–77. 6. ^ Mogey, Nora (March 25, 1999). "So You Want to Use a Likert Scale?". Learning Technology Dissemination Initiative. Heriot-Watt University. http://www.icbl.hw.ac.uk/ltdi/cookbook/info_likert_scale/index.html. Retrieved April 30, 2009. 7. ^ Babbie, Earl R. (2005). The Basics of Social Research. Belmont, CA: Thomson Wadsworth. p. 174. ISBN 0534630367. 8. ^ Meyers, Lawrence S.; Anthony Guarino, Glenn Gamst (2005). Applied Multivariate Research: Design and Interpretation. Sage Publications. p. 20. ISBN 1412904129. 9. ^ Latham, Gary P. (2006). Work Motivation: History, Theory, Research, And Practice. Thousand Oaks, Calif.: Sage Publications. p. 15. ISBN 0761920188.

[edit] External links •

Trochim, William M. (October 20, 2006). "Likert Scaling". Research Methods Knowledge Base, 2nd Edition.. http://www.socialresearchmethods.net/kb/scallik.php. Retrieved April 30, 2009.

•

Uebersax, John S. (2006). "Likert Scales: Dispelling the Confusion". http://www.johnuebersax.com/stat/likert.htm. Retrieved August 17, 2009.

•

"A search for the optimum scale". Get. http://www.get.net/kb/Choosing-the-optimium--scale.

•

Correlation scatter-plot matrix - for ordered-categorical data - On the visual presentation of correlation between Likert scale variables

Retrieved from "http://en.wikipedia.org/wiki/Likert_scale" Categories: Psychometrics | Educational psychology research methods | Market research | Questionnaire construction

Hidden categories: All articles with unsourced statements | Articles with unsourced statements from October 2009 | Articles with unsourced statements from December 2010 Personal tools

•

/ create

Namespaces

•

Article

•

Discussion

Variants Views

•

Read

•

Edit

•

View history

Actions Search Top of Form

Special:Search Search Bottom of Form

Navigation

•

Main page

•

Contents

•

Featured content

•

Current events

•

Random article

•

Donate to Wikipedia

Interaction

•

Help

•

About Wikipedia

•

Community portal

•

Recent changes

•

Wikipedia

Toolbox

•

What links here

•

Related changes

•

file

•

Special pages

•

Permanent link

•

Cite this page

Print/export

•

Create a book

•

as PDF

•

Printable version

Languages

•

‫العربية‬

•

Česky

•

Dansk

•

Deutsch

•

Español

•

Euskara

•

Français

•

한국어

•

Bahasa Indonesia

•

Italiano

•

Latviešu

•

Nederlands

•

日本語

•

Polski

•

Português

•

Русский

•

Српски / Srpski

•

Suomi

•

Svenska

•

Тоҷикӣ

•

中文

•

This page was last modified on 3 June 2011 at 07:57.

•

Text is available under the Creative Commons Attribution-ShareAlike License; additional may apply. See of Use for details. Wikipedia® is a ed trademark of the Wikimedia Foundation, Inc., a non-profit organization.

•

us

•



•

About Wikipedia

•

• •

Disclaimers

The value of Likert scales in measuring attitudes of online learners Hilary Page-Bucci - February 2003 Attitude is an important concept that is often used to understand and predict people's reaction to an object or change and how behaviour can be influenced (Fishbein and Ajzen, 1975) Introduction

Although online learning has grown alongside the progress of digital technology over the last 15 years; the reasoning behind why students become absorbed, practise and achieve a variety of tasks and exercises, or why they avoid others are always of interest to the effectors and evaluators of the learning process. By establishing the characteristics of distance and online learners; how they become motivated, how they feel about learning online; useful information will be found that would empower the teaching practices and thus ultimately enhance student retention and achievement. A review of some of the literature available has revealed some research already undertaken in various areas of learning online, such as 'training effectiveness and attitudes' (Torkzadeh et al, 1999). Torkzadeh et al suggest, " to achieve successful training we need to be cognizant of the 's attitudes towards computers. Further investigation revealed other factors that should be taken into consideration; Miltiadou (1999) suggests that 'it is important to identify motivational characteristics of online students'. By investigating and defining their motivation, it would lead to an understanding of 'self-efficacy beliefs about their own abilities to engage, persist and accomplish specific tasks' (Bandura, 1986; Stipek, 1988 cited by Miltiadou). The concept of measuring attitude is found in many areas including social psychology and the Social Sciences; they can be complex and difficult to measure and there are a number of different measuring instruments that have been developed to assess attitude. 'Scaling is the science of determining measuring instruments for human judgment' (McIver 1981). One needs to make use of appropriate scaling methods to aid in improving the accuracy of subjective estimation and voting procedures (Turoff & Hiltz 1997). Torgerson (1958) pointed out that scaling, as a science of measuring human judgment, is as fundamental as collecting data on well-developed natural sciences. Nobody would refute the fact that all science advances by the development of its measurement instruments. Researchers are constantly attempting to obtain more effective scaling methods that could be applied to the less well developed yet more complicated social sciences. Scaling models can be distinguished according to whether they are intended to scale persons, stimuli, or both (McIver 1981). For example, Likert scale is a subjectcentered approach since only subjects receive scale scores. Thurstone scaling is considered a method to evaluate the stimuli with respect to some designated attributes. It is the stimuli rather than the persons that are scaled (Togerson 1958). Guttman scaling is an approach in which both subjects and stimuli can be assigned scale values (McIver 1981). (Li et al, 2001) The purpose of this study is to explore the particular method of measuring attitude known as

Likert Scales (Likert, 1932), and determine their effectiveness and value in researching attitudes, views and experiences of online learners. These scales according to Taylor and Heath (1996) have become one of the dominant methods of measuring social and political attitudes. Methodology and Measurement

The methodology used for this research will be by a critique of previous research methodologies. In order to establish the methodology of this research it is first necessary to clarify the term 'attitude'. Attitude is an important concept that is often used to understand and predict people's reaction to an object or change and how behaviour can be influenced (Fishbein and Ajzen, 1975) An attitude is a mental and neural state of readiness, organised through experience, exerting a directive or dynamic influence upon the individual's response to all objects and situations to which it is related (Allport, 1935 cited by Gross) A learned orientation, or disposition, toward an object or situation, which provides a tendency to respond favourably or unfavourably to the object or situation.' (Rokeach, 1968 cited by Gross) Three of the generally accepted components of the term 'attitude' (Triandis, 1971) appear in some of the above definitions, these are: •

Affective - the person's feelings about the attitude object

•

Cognitive - the person's beliefs or knowledge about the attitude object

•

Behavioural - the person's inclination to act toward the attitude object on a particular way

By analysing these components, and as Gross (1968) suggests it is a 'hypothetical construct'; it becomes apparent that it cannot be directly measured and the use of only a single statement or question to assess it [attitude] will not be effective in gaining reliable responses. Attitude scales attempt to determine what an individual believes, perceives or feels. Attitudes can be measured toward self, others, and a variety of other activities, institutions, and situations (Gay, 1996) There are several types of scales that have been developed to measure attitude: Thurstone Scales

This is described by Thurstone & Chave (1929) as a method of equal-appearing intervals. Thurstone scalling is 'based on the law of comparative judgment' (Neuman, 2000). It requires the individual to either agree or disagree with a large number of statements about an issue or object. Thurstone scales typically present the reader with a number of statements to which they have to respond, usually by ticking a true/false box, or agree/disagree, i.e. a choice of two possible responses. Although one of the first scaling methods to be developed, the questionnaires are mostly generated by face to face interviews and rarely used in determining attitude measurement today, thus the example below (figure 1) is irrelevant to online learners. An example of a Thurstone Scale (figure1) ATTITUDE TOWARD WAR An individual is asked to check those items which represent his views. 1. A country cannot amount to much without a national honor, and war

is the only means of preserving it. 2. When war is declared, we must enlist. 3. Wars are justifiable only when waged in defense of weaker nations. 4. Peace and war are both essential to progress. 5. The most that we can hope to accomplish is the partial elimination of war. 6. The disrespect for human life and rights involved in a war is a cause of crime waves. 7. All nations should disarm immediately. (Droba, 1930) figure 1 Source: http://online.sfsu.edu/~psych200/unit8/84.htm

Advantages Disadvantages Items are weighted or More difficult to construct than a valued rather than subjects Likert scale Easier to construct than a Guttman scale

No more reliable than a Likert scale Measures only agreement or disagreement

Guttman Scales (Cumulative scales)

Guttman developed this scale in the 1940s in order to determine if a relationship existed within a group of items. The items are ordered from low to high according to difficulty so that to approve or correctly answer the last item implies approval or success of all prior ones (e.g. self-efficacy scale). The respondent selects an item that best applies. The list contains items that are cumulative, so the respondant either agrees or disagrees, if he/she agrees to one, he/she probably agrees to the previous statements. Arguably this scale does not give enough variation of feelings and perceptions, therefore the author suggests, this would not be appropriate for measuring attitude of online learners. An example of a Guttman Scale (figure 2): Please indicate what you think about new information technology (IT) by ticking ONE box to identify the statement that most closely matches your opinion (Wilson, 1997)

IT has no place in the office.

Agree

IT needs experts to use it in the office. IT can be used in the office by those with training. I'd be happy to have someone use IT to do things for me in the office. I'd be happy to use IT if I was trained. I'd be happy to teach myself to use IT. figure 2: Source: http://www.hb.se/bhs/nyutb/kurswebb/c-kurser/applirm/qdes4.htm

Advantages Reproducibility

Disadvantages Difficult to construct

Scalogram analysis may be too More one-dimensional than restrictive, only a narrow Likert scaling universe of content can be used Cornell technique questionable Results no better than summated Likert scales Semantic Differential Scaling

This is concerned with the 'measurement of meaning', the idea or association that individuals attach to words or objects. The respondent is required to mark on a scale between two opposing opinions (bipolar adjectives) the position they feel the object holds on that scale for them. It is often used in market research to determine how consumers feel about certain products. Three main factors emerge from the ratings, these are: The evaluative factor (good-bad, pleasant-unpleasant, kind-cruel); the potency factor (strongweak, thick-thin, hard-soft); the activity factor (active-ive, slow-fast, hot-cold) (Osgood et al, 1957). Although this scale is comparatively easy for the respondent to complete, the author argues that this would not be suitable for measuring attitude of online learners as it tends to relate more to material associations than cognizance of feelings. An example of a Semantic Differential Scale (figure 3):

figure 3

Advantages Simple to construct

Disadvantages Analyses can be complex

Easy for subjects to answer Allows for several types of analyses to take place

Likert Scale (Summated scale)

This was developed by Rensis Likert in 1932. It requires the individuals to make a decision on their level of agreement, generally on a five-point scale (ie. Strongly Agree, Agree, Disagree, Strongly Disagree) with a statement. The number beside each response becomes the value for that response and the total score is obtained by adding the values for each response, hence the reason why they are also called 'summated scales' (the respondents score is found by summing the number of responses). Dumas (1999) suggests, ' this is the most commonly used question format for assessing participants' opinions of usability'. Two examples of Likert Scales (figures 4 & 5):

figure 4

figure 5

Advantages Simple to construct

Disadvantages Lack of reproducibility

Each item of equal value so that Absence of onerespondents are scored rather dimensionality or than items homogeneity Likely to produce a highly reliable Validity may be difficult to scale demonstrate Easy to read and complete Reliability and Validity

Likert scale measures are fundamentally at the ordinal level of measurement because responses indicate a ranking only. As the number of scale steps is increased from 2 up through 20, the increase in reliability is very rapid at first. It tends to level off at about 7, and after 11 steps, there is little gain in reliability from increasing the number of steps (Nunally, 1978, cited by Neuman) Interestingly, Dyer (1995) states, 'attitude scales do not need to be factually accurate - they simply need to reflect one possible perception of the truth. ……[respondents] will not be assessing the factual accuracy of each item, but will be responding to the feelings which the statement triggers in them' In line with the above statement, when constructing a Likert scale a pool of statements needs to be generated that are relevant to the attitude (not necessarily fact), (figure 6). The number of choices on the scale should be evenly balanced to retain a continuum of positive and negative statements with which the respondent is likely to agree or disagree although the actual number of choices can be increased. This will help avoid the problem of bias (figure 7) and improves reliability as anyone who answers 'agree' all the time will appear to answer inconsistently.

figure 6

figure 7

As early as 1967, Tittle et al suggest, The Likert Scale is the most widely used method of scaling in the social sciences today. Perhaps this is because they are much easier to construct and because they tend to be more reliable than other scales with the same number of items (Tittle et al, 1967) But there still seems to be some contention within research as to whether Likert Scales are a good instrument for measuring attitude; Gal et al (1994) suggest 'Likert-type scales reveal little about the causes for answers........it appears they have limited usefulness'. Helgeson (1993) states that major reviews 'repeatedly point to two problems: lack of conceptual clarity in defining attitudes.....technical limitations of the instrument used to assess attitude' (Helgeson, 1993 cited by Gal et al 1994). The author suggests that some of these 'major' reviews have taken place prior to 1993, and along with the progress in technology, the reasons for measuring attitude may have also changed. It should also be taken into that this type of scale is not developed to provide any kind of diagnostic information that shows underlying issues of concern to the individual respondents. There are so many questionnaires students are asked to complete in the course of their studies, the interface and usability should be taken into consideration. There are now also researchers who are in favour of using Likert Scales; Robson (1993) suggests, Likert Scales 'can look interesting to respondents and people often enjoy completing a scale of this kind. This means that answers are more likely to be considered rather than perfunctory; and Neuman (2000) who states, 'the simplicity and ease of use of the Likert scale is its real strength'. Reservations on the use of a central Neutral Point

Arguments exist for including and not including a neutral point, and it would be reasonable to ask what effect adding a neutral point has on the responses you receive. Is it possible that some respondents may be neutral? In which case it could be argued that by not including a neutral point in a scale, the respondent is compelled to make a decision. Kline (cited by Eysenck, 1998) argues for a middle point, 'even though some participants will very often opt out by remaining indecisive'. Differing with this opinion it has been suggested, the traditional idea suggests that the qualitative results between the two scales are unaffected since if the respondents are truly neutral, then they will randomly choose one or the other, so forcing them to choose should not bias the overall results (Kahn et al,2000) It is also suggested that the exclusion of a neutral point will draw the respondent to make a decision one way or the other. This, states Dumas (1999), 'means that by eliminating a neutral level it is providing a better measure of the intensity of participants' attitudes or opinions'. The author suggests that by preventing the respondent to remain neutral, thus causing them to either 'agree' or 'disagree' could reduce the reliability of the scale as the results will not necessarily be true. Review of Literature

Torkzadeh et al (2001) describe the construction of a scale to measure an individual's selfperception and self-competency in interacting with the Internet. They consulted five practitioners and four academics and developed a five-point Likert-type scale (where 1 is strongly disagree to

5 is strongly agree) using a list of 24 items with objectives to explore responses relating to 'unidimensionality, reliability, brevity and simplicity of the factor structure'. The survey was istered at a university in the Southwest region of the United States to a total of 227 students, with an age range from 17 to 57 years. They used two main criteria for eliminating items that were not considered valid and reliable; firstly if the correlation of each item with the sum of the other items in its category was less than 0.50. This was using the assumption that 'if all items in a measure are drawn from the domain of a single construct, responses to those items should be highly intercorrelated'. The second criterion was for determining reliability; Cronbach's alpha was used to examine each dimension to see if 'additional items could be eliminated without substantially lowering the reliability'. 'Items were eliminated if the reliability of the remaining items would be at least 0.90.' The resulting figures showed evidence of reliability and construct validity, overall reliability for the scale had a coefficient alpha reliability score of 0.96. The final recommendation after taking into consideration that this was their first exploratory model stated; 'the instrument should also be validated across other variables such as age, education level and profession in order to assess the generalisability of the scale to a more heterogeneous population' but this was not a reflection of the instrument itself. In conclusion, they stated the 'instrument is useful in its present form' although one must always be aware of the ever changing technologies on the World Wide Web and the need to keep up to date with progress. "this instrument is short, easy to use, reliable and appropriate for use by academics and practitioners to measure Internet-related self-efficacy." (Torkzadeh et al, 2001) Shaw et al (2000) used a questionnaire arranged in a Likert format to determine attitudes, views and experiences of a group of nutrition students using an asynchronous learning network. The data was obtained through an online 'IT Appreciation' questionnaire completed in class during week 12 of the course. 'The text match questions allowed students to express opinions in their own words and the multiple choice format consisted of 5 possible responses (some reversed to counteract response sets) to the given statement arranged in a Likert format' (Shaw et al, 2000). It was concluded that the ALN paradigm could be considered a success as the majority of the respondents agreed with the statements that they had become more independent learners. But it was also noted that the largely positive responses to the Likert questions were contradicted by the student responses to the open ended questions. From this it was decided that further study should determine the discrepancy between the responses to the Likert question and the open-ended questions; it was also considered a possibility that this could be due to the Hawthorne effect (behaviour may be altered because the respondents know they are being studied.) The author suggests therefore, that although the questionnaire was considered a success, the initial construction of the questionnaire along with how it is presented (i.e. online in the classroom with other students or away from the class situation) needs to be considered carefully. The apparent acquiescence could be because the questions some of the questions were single-sided, (although it was stated otherwise) or perhaps there was a large number of 'don't knows' or 'non-responses'; the results don't include any information on this. Rovai (2002) used a Likert-type scale, referred to as the 'Classroom Community Scale' in his study of 314 distance learners using Blackboard as the mode of delivery. The research was 'to determine if a significant relationship exists between sense of community and cognitive learning in an online educational environment'; with the premise that if online learners feel an 'emotional

connectedness' to a community, their learning and motivation will be increased. 20 statements were used (some reverse scored), with a five-point scale of responses: strongly agree, agree, neutral, disagree and strongly disagree. Cronbach's coefficient alpha was used to calculate the reliability which was .93. Content validity was examined by a of experts comprising three university professors of educational psychology. Although there is an in-depth discussion with regard to further research, and assumptions that the respondents were typical students that participate in online distance education, the overall conclusion showed that the Classroom Community Scale 'allowed for the hypothesised relationships between the sense of community and cognitive learning'. The author suggests, this Likert-type scale which has been adapted and renamed shows there is considerable scope for the use of Likert scales in an e-learning environment. Conclusions

Moving questionnaires with Likert scales onto the World Wide Web brings a whole new meaning to questionnaires. They could almost be another source of activity for the online learner. A form of scale that is frequently used is the 'graphic scale', the respondent indicates his/her rating by placing a mark at the appropriate point on a line that runs from one extreme of the attribute to the other. To be a true Likert scale after the series of items has been developed using a graphic rating scale, it is then necessary to determine which items have the highest correlation with a specific criterion measure; only these will be included in the scale. Although not a graphic scale, figure 8 shows an example of how a Likert scale could be presented in a web page. The use of radio buttons makes it easy to complete, and as there is only one choice, difficult to invalidate by ticking two boxes. Top of Form

It was easy for me to how to perform tasks using spreadsheets

Strongly Disagree

Disagree

Neither

Agree

Strongly Agree

Bottom of Form

figure 8

Other methods of presenting Likert scales in a web page are by using slider controls (figures 9, 10, 11 & 12). A “slider control” (also known as a trackbar) is a window containing a slider and optional tick marks. They are useful when you want the respondent to select a discrete value or a set of consecutive values in a range. When the moves the slider, using either the mouse or the direction keys, the control sends notification messages to indicate the change.

figure 9 Source: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/shellcc/platform/commctls/trackbar/trackbar.asp

figure 10 Source: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/shellcc/platform/commctls/trackbar/trackbar.asp

figure 11 Source: http://archive.devx.com/dhtml/articles/nm061102/slider.html

figure 12 Source: http://archive.devx.com/dhtml/articles/nm061102/Hand.html

The slider moves in increments that you specify when you create it. For example, if you specify that the slider should have a range of five, the slider can only occupy six positions: a position at the left side of the slider control and one position for each increment in the range. From a technical aspect, a basic knowledge of programming is useful if the designer of the survey or questionnaire wishes to include slider controls in a web page. Radio buttons (figure 8) require a knowledge of html making them an easier option for the less technically minded. Although there is some question of the reliability of Likert scales and their analytical capacity, the general consensus is in favour of using Likert scales; this is reinforced by the majority of the latterly dated literature reviewed. Maurer and Pierce (cited by Maurer and Andrews, 2000) investigated the effectiveness of a Likert scale measure of self-efficacy for academic performance. They suggested the Likert scale can be considered a measure of both magnitude and confidence, and they concluded, based on

reliability, predictive validity, and factor analysis data, that a Likert scale measure of selfefficacy is an acceptable alternative to the traditional measure. Bibliography

Bell, J (1999) Doing Your Research Project. Buckingham: Open University Press Bouma, G & Atkinson, G (1995) A Handbook of Social Science Research. New York: Oxford University Press Cohen, L; Manion, L; & Morrison, K (2000) Research Methods in Education. London: RoutledgeFalmer Coolican, H (1995) Introduction to Research Methods and Statistics in Psychology. London; Hodder and Stoughton Coolican, H (1990) Research Methods and Statistics in Psychology. London; Hodder and Stoughton Dumas, J (1999) Usability Testing Methods: Subjective Measures, Part II - Measuring Attitudes and Opinions. American Institutes for Research. Available online: http://www.uoc.org/html/1999_archive/usability_testing_methods.html [20.02.03] Dyer, C (1995) Beginning Research in Pschology. Oxford: Blackwell Dwyer, E (1993) Attitude Scale Construction: A Review of the Literature. Abstract Available online: http://www.askeric.org Document. no. ED359201 [16.01.03] Eysenck, M (1998) Psychology: An Integrated Approach. Essex; Addison Wesley Longman Fishbein, M & Ajzen, I (1975) Belief, Attitude, Intention and Behaviour: An Introduction to Theory and Research. London: Addison-Wesley Gal, I & Ginsburg, L (1994) The Role of Beliefs and Attitudes in Learning Statistics: Towards an Assessment Framework. Journal of Statistics Education, Vol. 2, No. 2 Available online: http://www.amstat.org/publications/jse/v2n2/gal.html [16.02.03] Gall, M; Borg, W & Gall, J (1996) Educational Research, An Introduction. (6th Edition). USA: Longman Gay, L (1996) Educational Research: Competencies for Analysis and Application. New Jersey: Prentice Hall Gross, R (2001) Psychology: The Science of Mind and Behaviour. London; Hodder and Stoughton Kahn, B; Nowlis, S; & Dhar, R Indifference versus Ambivalence: The Effect of a Neutral Point on Consumer Attitude and Preference Measurement. Available online: http://hops.wharton.upenn.edu/ideas/pdf/00-022.pdf [22.02.03] Likert, R (1932) A Technique for the Measurement of Attitudes. Archives of Psychology; No.140 Li, Z; Cheng, K; Wang, Y; Hiltz, S; Turoff, M; (2001) Thurstones Law of Comparative Judgment for Group . Available online: http://web.njit.edu/~zxl8078/Publication/BB004.PDF [16.02.03] Likert, R & Hayes, S (1957) Some Applications of Behavioural Research. Paris: Unesco

Maurer, T & Andrews, K (2000) Traditional, Likert and Simplified Measures of Self-efficacy. Educational and Pschological Measurement; Vol. 60, No. 6 Miltiadou, M (1999) Motivational Constructs as Predictors of success in the Online Classroom. Available online: http://seamonkey.ed.asu.edu/~mcisaac/emc703/mariosf.html [7.02.03] Miltiadou, M (1998?) Validation of the Online Technologies Self-efficacy Scale. Available online: http://seamonkey.ed.asu.edu/~alex/pub/efficacy.pdf [7.02.03] Moser, C & Kalton G (1989) Survey Methods in Social Investigation. Hampshire, UK: Gower Publishing Neuman, W.L (2000) Social Research Methods: Qualitative and Quantitative Approaches. USA: Allyn & Bacon Newell, S & Goldsmith, R (2001) The Development of a scale to measure Perceived Corporate Credibility. Journal of Business Research; Vol. 52, Issue 3 Osgood, E; Suci, G & Tannenbaum, P (1957) The measurement of Meaning. Urbana: University of Illinois Robson, C (1993) Real World Research: A Resource for Social Scientists and PractitionerResearchers. Oxford: Blackwell Publishers Rovai, A (2002) Sense of Community, perceived cognitive learning, and persistence in asynchronous learning networks. The Internet and Higher Education; Vol. 5; Issue 4 Shaw, G & Pieter, W (2000) The Use of Asynchronous Learning Networks in Nutrition Education: Student Attitude, Experiences and Performance. Available online: http://www.aln.org/publications/jaln/v4n1/pdf/v4n1_shawpieter.pdf [24.01.03] Taylor, B & Heath, A (1996) The Use of Double-sided Items in Scale Construction. Centre for Research into Elections and Social Trends; Working Paper no. 37. Abstract available online: http://www.crest.ox.ac.uk/p37.htm [11.01.03] Tittle, C & Hill, R (1967) Attitude Measurement and Prediction of Behaviour: An Evaluation of Conditions and Measurement Techniques. Sociometry, Vol. 30 Torkzadeh, R; Pflugheoft, K & Hall, L (1999) Computer self-efficacy training effectiveness and attitudes: an empirical study. Behaviour & Information Technology vol. 18, no 4 Torkzadeh, G & Van Dyke, T (2001) Development and validation of an Internet self-efficacy scale. Behaviour and Information Technology, vol. 20, no 4 Wilson, T (1997) Online Course on Questionnaire Design. Available online: http://www.hb.se/bhs/nyutb/kurswebb/c-kurser/applirm/qdes4.htm [20.02.03]

Chapter 6 Methods of Data Collection (Note: For the concept map that goes with this lecture, click here. : concept maps help provide the big picture as well as show how the parts are interrelated.) The purpose of Chapter 6 is to help you to learn how to collect data for a research project. • The term method of data collection simply refers to how the researcher obtains the empirical data to be used to answer his or her research questions. • Once data are collected they are analyzed and interpreted and turned into information and results or findings. • All empirical research relies on one or more method of data collection. It is important to consider and utilize the fundamental principle of mixed research during the planning of a research study. • The principle states that researchers should mix methods (including methods of data collection as well as methods of research) in a way that is likely to provide complementary strengths and nonoverlapping weaknesses.

•

•

We will provide you with additional tables (not in the chapter because of space limitations) for each method of data collection so that you can compare the strengths and weaknesses of each method of data collection and attempt to put together the match that will best serve your purpose and will follow the fundamental principle of mixed research. The focus in this chapter is on methods of data collection, not methods of research (which are covered in later chapters).

There are six major methods of data collection. We will briefly summarize each of these in this lecture: •

Tests (i.e., includes standardized tests that usually include information on reliability, validity, and norms as well as tests constructed by researchers for specific purposes, skills tests, etc).

•

Questionnaires (i.e., self-report instruments).

•

Interviews (i.e., situations where the researcher interviews the participants).

•

Focus groups (i.e., a small group discussion with a group present to keep the discussion focused).

•

Observation (i.e., looking at what people actually do).

•

Existing or Secondary data (i.e., using data that are originally collected and then archived or any other kind of “data” that was simply left behind at an earlier time for some other purpose).

Tests Tests are commonly used in research to measure personality, aptitude, achievement, and performance. The last chapter discussed standardized tests; therefore, we only have a brief discussion in this chapter. Note that tests can also be used to complement other measures (following the fundamental principle of mixed research). In addition to the tests discussed in the last chapter, note that sometimes, a researcher must develop a new test to measure the specific knowledge, skills, behavior, or cognitive activity that is being studied. For example, a researcher might need to measure response time to a memory task using a mechanical apparatus or develop a test to measure a specific mental or cognitive activity (which obviously cannot be directly observed). • • An excellent source of tests (and other measures) (that we didn’t get into the chapter in time) is called The Directory of Unpublished Experimental Mental Measures (2003) edited by Goldman and Mitchell, published by the American Psychological Association. • • We list the major sources of tests and test reviews in Table 5.7. • • We listed the major internet sources for finding tests in Table 5.8 • • that if a test has already been developed that purports to measure what you want to measure, then you should strongly consider using it rather.

The following table lists the strengths and weaknesses of tests. It, in conjunction with the tables for the other five major methods of data collection, will help you in applying the fundamental principle of mixed research: Strengths and Weaknesses of Tests Strengths of tests (especially standardized tests) • Can provide measures of many characteristics of people. • Often standardized (i.e., the same stimulus is provided to all participants). • Allows comparability of common measures across research populations. • Strong psychometric properties (high measurement validity). • Availability of reference group data. • Many tests can be istered to groups which saves time. • Can provide “hard,” quantitative data. • Tests are usually already developed. • A wide range of tests is available (most content can be tapped). • Response rate is high for group istered tests. • Ease of data analysis because of quantitative nature of data. Weaknesses of tests (especially standardized tests) • Can be expensive if test must be purchased for each research participant. • Reactive effects such as social desirability can occur. • Test may not be appropriate for a local or unique population. • Open-ended questions and probing not available. • Tests are sometimes biased against certain groups of people. • Nonresponse to selected items on the test. • Some tests lack psychometric data.

Questionnaires A questionnaire is a self-report data collection instrument that is filled out by research participants. Questionnaires are usually paper-and-pencil instruments, but they can also be placed on the web for participants to go to and “fill out.” Questionnaires are sometimes called survey instruments, which is fine, but the actual questionnaire should not be called “the survey.” The word “survey” refers to the process of using a questionnaire or interview protocol to collect data. For example, you might do a survey of teacher attitudes about inclusion; the instrument of data collection should be called the questionnaire or the survey instrument. • A questionnaire is composed of questions and/or statements. • Because one way to learn to write questionnaires is to look at other questionnaires, here is an example of a typical questionnaire that has mostly quantitative items, click here. • For an example of a qualitative questionnaire, click here. • When developing a questionnaire make sure that you follow the 15 Principles of Questionnaire Construction. I will briefly review the 15 principles now. Principle 1: Make sure the questionnaire items match your research objectives. Principle 2: Understand your research participants. • Your participants (not you!) will be filling out the questionnaire.

•

Consider the demographic and cultural characteristics of your potential participants so that you can make it understandable to them.

Principle 3: Use natural and familiar language. • Familiar language is comforting; jargon is not. Principle 4: Write items that are clear, precise, and relatively short. • If your participants don't understand the items, your data will be invalid (i.e., your research study will have the garbage in, garbage out, GIGO, syndrome). • Short items are more easily understood and less stressful than long items. Principle 5: Do not use "leading" or "loaded" questions. • Leading questions lead the participant to where you want him or her to be. • Loaded questions include loaded words (i.e., words that create an emotional reaction or response by your participants). • Always that you do not want the participant's response to be the result of how you worded the question. Always use neutral wording. Principle 6: Avoid double-barreled questions. • A double-barreled question combines two or more issues in a single question (e.g., here is a double barreled question: “Do you elicit information from parents and other teachers?” It’s double barreled because if someone answered it, you would not know whether they were referring to parents or teachers or both). • Does the question include the word "and"? If yes, it might be a double-barreled question. • Answers to double-barreled questions are ambiguous because two or more ideas are confounded. Principle 7: Avoid double negatives. • Does the answer provided by the participant require combining two negatives? (e.g., "I disagree that teachers should not be required to supervise their students during library time"). If yes, rewrite it. Principle 8: Determine whether an open-ended or a closed ended question is needed. • Open-ended questions provide qualitative data in the participants' own words. Here is an open ended question: How can your principal improve the morale at your school? _______________________________________________ • Closed-ended questions provide quantitative data based on the researcher's response categories. Here is an example of a closed-ended question:

•

Open-ended questions are common in exploratory research and closed-ended questions are common in confirmatory research.

Principle 9: Use mutually exclusive and exhaustive response categories for closed-ended questions. • Mutually exclusive categories do not overlap (e.g., ages 0-10, 10-20, 20-30 are NOT mutually exclusive and should be rewritten as less than 10, 10-19, 20-29, 30-39, ...). • Exhaustive categories include all possible responses (e.g., if you are doing a national survey of adult citizens (i.e., 18 or older) then the these categories (18-19, 20-29, 30-39, 40-49, 50-59, 60-69) are NOT exhaustive because there is no where to put someone who is 70 years old or older. Principle 10: Consider the different types of response categories available for closed-ended questionnaire items. • Rating scales are the most commonly used, including: ○

1 Very Low ○

Numerical rating scales (where the endpoints are anchored; sometimes the center point or area is also labeled). 2

3

4

5

6

7 Very High

Fully anchored rating scales (where all the points on the scale are anchored).

1 Strongly Agree

2 Agree

3 Neutral

4 Disagree

1 Strongly Agree

2 Agree

3 Disagree

4 Strongly Disagree

5 Strongly Disagree

○

Omitting the center point on a rating scale (e.g., using a 4-point rather than a 5point rating scale) does not appreciably affect the response pattern. Some researchers prefer 5- point rating scales; other researchers prefer 4-point rating scales. Both generally work well.

○

You should use somewhere from four to eleven points on your rating scale. Personally, I like the 4 and 5-point scales because all of the points are easily anchored.

○ I do not recommend a 1 to 10 scale because too many respondents mistakenly view the 5 as the center point. If you want to use a wide scale like this, use a 0 to

10 scale (where the 5 is the middle point) and label the 5 with the anchor “medium” or some other appropriate anchor. •

Rankings (i.e., where participants put their responses into rank order, such as most important, second most important, and third most important).

•

Semantic differential (i.e., where one item stem and multiple scales, that are anchored with polar opposites or antonyms, are included and are rated by the participants).

•

Checklists (i.e., where participants "check all of the responses in a list that apply to them").

Principle 11: Use multiple items to measure abstract constructs. • This is required if you want your measures to have high reliability and validity. • One approach is to use a summated rating scale(such as the Rosenberg Self-Esteem Scale that is composed of 10 items, with each item measuring self-esteem). • Another name for a summated rating scale is a Likert Scale because the summated rating scale was pretty much invented by the famous social psychologist named Rensis Likert. • Here is the Rosenberg Self-Esteem Scale, which is a summated rating scale:

Principle 12: Consider using multiple methods when measuring abstract constructs. • The idea here is that if you only use one method of measurement, then your measurement may be an artifact of that method of measurement. • On the other hand, if you use two or more methods of measurement you will be able to see whether the answers depend on the method (i.e., are the answers corroborated across the methods of measurement or do you get different answers for the different methods?). For example, you might measure student’s self-esteem via the Rosenberg Scale just shown (which is used in a self-report form) as well as using teachers’ ratings of the students’ self-esteem; you might even want to observe the students in situations that should provide indications of high and low self-esteem.

Principle 13: Use caution if you reverse the wording in some of the items to prevent response sets. (A response set is the tendency of a participant to respond in a specific direction to items regardless of the item content.) • Reversing the wording of some items can help ensure that participants don't just "speed through" the instrument, checking "yes" or "strongly agree" for all the items. • On the other hand, you may want to avoid reverse wording if it creates a double negative. • Also, recent research suggests that the use of reverse wording reduces the reliability and validity of scales. Therefore, you should generally use reverse wording sparingly, if at all. Principle 14: Develop a questionnaire that is easy for the participant to use. • The participant must not get confused or lost anywhere in the questionnaire. • Make sure that the directions are clear and that any filter questions used are easy to follow. Principle 15: Always pilot test your questionnaire. • You will always find some problems that you have overlooked! • The best pilot tests are with people similar to the ones to be included in your research study. • After pilot testing your questionnaire, revise it and pilot test it again, until it works correctly. The following table lists the strengths and weaknesses of questionnaires. It, in conjunction with the tables for the other five major methods of data collection, will help you in applying the fundamental principle of mixed research: Strengths and Weaknesses of Questionnaires Strengths of questionnaires • Good for measuring attitudes and eliciting other content from research participants. • Inexpensive (especially mail questionnaires and group istered questionnaires). • Can provide information about participants’ internal meanings and ways of thinking. • Can ister to probability samples. • Quick turnaround. • Can be istered to groups. • Perceived anonymity by respondent may be high. • Moderately high measurement validity (i.e., high reliability and validity) for well constructed and validated questionnaires. • Closed-ended items can provide exact information needed by researcher. • Open-ended items can provide detailed information in respondents’ own words. • Ease of data analysis for closed-ended items. • Useful for exploration as well as confirmation. Weaknesses of questionnaires • Usually must be kept short. • Reactive effects may occur (e.g., interviewees may try to show only what is socially desirable). • Nonresponse to selective items. • People filling out questionnaires may not recall important information and may lack selfawareness. • Response rate may be low for mail and email questionnaires.

• • •

Open-ended items may reflect differences in verbal ability, obscuring the issues of interest. Data analysis can be time consuming for open-ended items. Measures need validation.

Interviews In an interview, the interviewer asks the interviewee questions (in-person or over the telephone). • Trust and rapport are important. • Probing is available (unlike in paper-and-pencil questionnaires) and is used to reach clarity or gain additional information • Here are some examples of standard probes: - Anything else? - Any other reason? - What do you mean? Interviews may be quantitative or qualitative. Quantitative interviews: • Are standardized (i.e., the same information is provided to everyone). • Use closed-ended questions. • Exhibit 6.3 has an example of an interview protocol. Note that it looks very much like a questionnaire! The key difference between an interview protocol and a questionnaire is that the interview protocol is read by the interviewer who also records the answers (you have probably participated in telephone surveys before...you were interviewed). Qualitative interviews • They are based on open-ended questions. • There are three types of qualitative interviews. 1) Informal Conversational Interview. - It is spontaneous. - It is loosely structured (i.e., no interview protocol us used). 2) Interview Guide Approach. • It is more structured than the informal conversational interview. • It includes an interview protocol listing the open-ended questions. • The questions can be asked in any order by the interviewer. • Question wording can be changed by the interviewer if it is deemed appropriate. 3) Standardized Open-Ended Interview. • Open-ended questions are written on an interview protocol, and they are asked in the exact order given on the protocol. • The wording of the questions cannot be changed. The following table lists the strengths and weaknesses of interviews. It, in conjunction with the tables for the other five major methods of data collection, will help you in applying the fundamental principle of mixed research:

Strengths and Weaknesses of Interviews Strengths of interviews • Good for measuring attitudes and most other content of interest. • Allows probing and posing of follow-up questions by the interviewer. • Can provide in-depth information. • Can provide information about participants’ internal meanings and ways of thinking. • Closed-ended interviews provide exact information needed by researcher. • Telephone and e-mail interviews provide very quick turnaround. • Moderately high measurement validity (i.e., high reliability and validity) for well constructed and tested interview protocols. • Can use with probability samples. • Relatively high response rates are often attainable. • Useful for exploration as well as confirmation. Weaknesses of interviews • In-person interviews usually are expensive and time consuming. • Reactive effects (e.g., interviewees may try to show only what is socially desirable). • Investigator effects may occur (e.g., untrained interviewers may distort data because of personal biases and poor interviewing skills). • Interviewees may not recall important information and may lack self-awareness. • Perceived anonymity by respondents may be low. • Data analysis can be time consuming for open-ended items. • Measures need validation.

Focus Groups A focus group is a situation where a focus group keeps a small and homogeneous group (of 6-12 people) focused on the discussion of a research topic or issue. • Focus group sessions generally last between one and three hours and they are recorded using audio and/or videotapes. • Focus groups are useful for exploring ideas and obtaining in-depth information about how people think about an issue. The following table lists the strengths and weaknesses of focus groups. It, in conjunction with the tables for the other five major methods of data collection, will help you in applying the fundamental principle of mixed research: Strengths and Weaknesses of Focus Groups Strengths of focus groups • Useful for exploring ideas and concepts. • Provides window into participants’ internal thinking. • Can obtain in-depth information. • Can examine how participants react to each other. • Allows probing. • Most content can be tapped. • Allows quick turnaround. Weaknesses of focus groups • Sometimes expensive. • May be difficult to find a focus group with good facilitative and rapport building skills. • Reactive and investigator effects may occur if participants feel they are being watched or studied.

• • • • • •

May be dominated by one or two participants. Difficult to generalize results if small, unrepresentative samples of participants are used. May include large amount of extra or unnecessary information. Measurement validity may be low. Usually should not be the only data collection methods used in a study. Data analysis can be time consuming because of the open-ended nature of the data.

Observation In the method of data collection called observation, the researcher observes participants in natural and/or structured environments. • It is important to collect observational data (in addition to attitudinal data) because what people say is not always what they do! Observation can be carried out in two types of environments: • Laboratory observation (which is done in a lab set up by the researcher). • Naturalistic observation (which is done in real-world settings). There are two important forms of observation: quantitative observation and qualitative observation. 1) Quantitative observation involves standardization procedures, and it produces quantitative data. • The following can be standardized: - Who is observed. - What is observed. - When the observations are to take place. - Where the observations are to take place. - How the observations are to take place. • Standardized instruments (e.g., checklists) are often used in quantitative observation. • Sampling procedures are also often used in quantitative observation: --Time-interval sampling (i.e., observing during time intervals, e.g., during the first minute of each 10 minute interval). --Event sampling (i.e., observing after an event has taken place, e.g., observing after teacher asks a question). 2) Qualitative observation is exploratory and open- ended, and the researcher takes extensive field notes. The qualitative observer may take on four different roles that make up a continuum: • • Complete participant (i.e., becoming a full member of the group and not informing the participants that you are studying them). • • Participant-as-Observer (i.e., spending extensive time "inside" and informing the participants that you are studying them). • • Observer-as-Participant (i.e., spending a limited amount of time "inside" and informing them that you are studying them). • • Complete Observer (i.e., observing from the "outside" and not informing that participants that you are studying them).

The following table lists the strengths and weaknesses of observational data. It, in conjunction with the tables for the other five major methods of data collection, will help you in applying the fundamental principle of mixed research: Strengths and Weaknesses of Observational Data Strengths of observational data • Allows one to directly see what people do without having to rely on what they say they do. • Provides firsthand experience, especially if the observer participates in activities. • Can provide relatively objective measurement of behavior (especially for standardized observations). • Observer can determine what does not occur. • Observer may see things that escape the awareness of people in the setting. • Excellent way to discover what is occurring in a setting. • Helps in understanding importance of contextual factors. • Can be used with participants with weak verbal skills. • May provide information on things people would otherwise be unwilling to talk about. • Observer may move beyond selective perceptions of people in the setting. • Good for description. • Provides moderate degree of realism (when done outside of the laboratory). Weaknesses of observational data • Reasons for observed behavior may be unclear. • Reactive effects may occur when respondents know they are being observed (e.g., people being observed may behave in atypical ways). • Investigator effects (e.g., personal biases and selective perception of observers) • Observer may “go native” (i.e., over-identifying with the group being studied). • Sampling of observed people and settings may be limited. • Cannot observe large or dispersed populations. • Some settings and content of interest cannot be observed. • Collection of unimportant material may be moderately high. • More expensive to conduct than questionnaires and tests. • Data analysis can be time consuming.

Secondary/Existing Data Secondary data (i.e., data originally used for a different purpose) are contrasted with primary data (i.e., original data collected for the new research study). The most commonly used secondary data are documents, physical data, and archived research data. 1. Documents. There are two main kinds of documents. • • Personal documents (i.e., things written or recorded for private purposes). Letters, diaries, family pictures. • • Official documents (i.e., things written or recorded for public or private organizations). Newspapers, annual reports, yearbooks, minutes. 2. Physical data (are any material thing created or left by humans that might provide information about a phenomenon of interest to a researcher).

3. Archived research data (i.e., research data collected by other researchers for other purposes, and these data are save often in tape form or cd form so that others might later use the data). For the biggest repository of archived research data, click here. The following table lists the strengths and weaknesses of secondary/existing data. It, in conjunction with the tables for the other five major methods of data collection, will help you in applying the fundamental principle of mixed research: Strengths and Weaknesses of Secondary Data Strengths of documents and physical data: • Can provide insight into what people think and what they do. • Unobtrusive, making reactive and investigator effects very unlikely. • Can be collected for time periods occurring in the past (e.g., historical data). • Provides useful background and historical data on people, groups, and organizations. • Useful for corroboration. • Grounded in local setting. • Useful for exploration. Strengths of archived research data: • Archived research data are available on a wide variety of topics. • Inexpensive. • Often are reliable and valid (high measurement validity). • Can study trends. • Ease of data analysis. • Often based on high quality or large probability samples. Weaknesses of documents and physical data: • May be incomplete. • May be representative only of one perspective. • Access to some types of content is limited. • May not provide insight into participants’ personal thinking for physical data. • May not apply to general populations. Weaknesses of archived research data: • May not be available for the population of interest to you. • May not be available for the research questions of interest to you. • Data may be dated. • Open-ended or qualitative data usually not available. • Many of the most important findings have already been mined from the data.