Evaluation Research Articles

Evaluation Research Articles

Empirical Research

The based of empirical research should include controlled observation, which are Reliability and Validity. For the Controlled observation, the researcher should:

• Control the conditions when data are collected
 Noise is eliminated/ minimized
 Gathering data under uniform conditions (“Superfluous factors”)
 Every participant has the same condition (“Environmental distraction”)

These features ensure that the collected data will be reliability and validity. For Reliability, it can refer that the researcher should:

 Under that same experimental conditions anyone should be able to obtain the same results
 Also refers to precision of our measuring instruments

Data Scores include two parts that are Systematic variance and error variance as the below formula.

Data Scores = Systematic variance + nonsystematic or error variance

There are several reliability test procedures that are:

 Test-Retest: The same group of people is given the test on 2 occasion- The scores will be the same in each time
 Split-half: Compare the 1st half vs 2nd half- if all items measure the same characteristic, there should be a relationship or correlation between the scores on each half of test
 Cronbach’s alpha: measures of reliability are in the form of correlation coefficient that ranges from 0.00 to .99 (at least .65)

Besides the reliability, there is another factor that is Validity. For the validity, it refers to the conclusions are valid, meaningful, and useful. There are several Validity Testing procedures that are:

• Content Validity: refers to the extent to which all items reflect the characteristic
• Criterion Validity: Split participant as the test scores that correlate with a behaviour
• Predictive Validity: Predict behaviour or performance as test score
• Construct Validity: The characteristic cannot be measured directly (Construct Validity is the most difficult to establish)

However, to ensure that the conclusions are valid, the researcher needs to avoid the threats of Validity. Validity can divide into 2 types that are Internal Validity and Statistical Conclusion Validity.

For Internal Validity, there are 2 types of the qualitative studies that are “Studies with pre- and posttests” and “Studies with or without pre-and posttests”. Each study gets different threats as the following:

 Studies with pretests and posttests

 History: Directly or indirectly could affect the behaviour
 Maturation: Refers to any changes within the participant that occurs during the test
 Instrumentation: Refers to any changes in the measuring instrument from pre- to posttest
 Initial Testing: A change in posttest performance that results from pretest experience
 Regression toward the mean: When pretest scores were extremely high or low, posttest scores are predicted to be less extreme, regardless of treatment effects

 Studies with or without pre- and posttests

 Selection bias: Difference in performance may be associated with a participant characteristic
 Selective loss: Loss of particular participants from a group
 Diffusion of treatment: The unintentional spread of treatment to a control group
 Compensatory rivalry: Participants receive unequal treatment
 Resentful Demoralization: Participants resent the lack of experimental treatment
 Experimenter expectancy: Participants behave as the hypotheses

For Statistical Conclusion Validity, there are 2 types of well-known errors that are called “Type I” and “Type II” errors.

• Type I: Reject the Null Hypothesis when it is true
• Type II: Retain the Null Hypothesis when it is not true

False True
Reject H0 OK Type I
Accept H0 Type II OK

There are a lot of reasons that cause “Type I” and “Type II” errors that are:

 Insufficient power of statistical test: : If samples are very large, any effect can be significant, and this could lead to a type I error
 Unreliable instrument: If the measuring instrument is not reliable, performance will be variable
 Varied test conditions: If testing conditions are not uniform, performances will be variable
 Varied participant characteristics: If participants have different characteristics, performances will be variable
 Violation of statistical assumption: If at least one is seriously violently, the analyses may fail to reveal a difference
 Fishing: When an unreasonable number of statistical tests are conducted on the same data, one may reveal what appears to be a significant difference

For the Empirical research, the reader should keep understand the article step by step as the following:

 Rationale: The past literature aroused the investigator’s interest in the subject
 Purpose: let you know what exactly the investigators intended to accomplish. Often, it is expressed in terms of hypothesis testing
 Method: illustrate how they were recruited
 General Procedure: what was done to the participants or what they were required to do
 Results: Appropriate analyses of the data
 Discussion: Conclusion regarding the outcome of their manipulation

Case studies

Case Studies appear in all of the social sciences and are examples of qualitative research. They range from those that are purely descriptive, explaining behaviour. In all cases, data can gathered by Observation, Interview and examination of documents. However, there are the Caution factors in Case Studies that are:

 The reason the case was selected
 The nature of intervention
 The objectivity of behavioural assessment
 The assessor of behaviour
 The past and/or present factors that could change the behaviour
 The method of selecting participants in a singe unit
 The reliability and validity of interview questions
 Investigator bias

Narrative Analysis

Many qualitative methods exist besides the case study. Narrative Analysis is one of them. The researcher will observe group interactions by joining the group. It also includes a temporal sequence of experiences, some themes running through them and structural coherence. In this analysis, most of the interview questions are generally open-ended.

There are 5 aspects to a narrative report of one’s experiences of an event:

 Attention to certain phenomena make them meaningful
 Narration is the telling of the event in the form of a story
 How the story is told depends on the listener- narratives about the same event will differ when the listeners are friends, colleagues or oversees
 Transcribers typically order narrative's experiences in sequence
 Analysis of the transcribed tape- involves decision about what will be included and excluded according to the researcher’s anticipated responses

However, different readers will attribute different meaning to the text depending on their personal experiences and expectations.

The core of the analysis is Trustworthiness of the interpretation, which will depend on:

 The nature of the questions asked in the interview
 The place in which the interview occurred
 A different type of analysis may lead to a different interpretation

For that reason, the Caution Factors in Narrative Analysis are:

 The reason the particular individual was interviewed
 The location in which the interview took place
 The nature of the interview questions
 The relationship between the interviewer and narrator

Survey

Surveys are almost always quantitative method to the population of interest. They are conducted with the intent of obtaining a general sense of what people feel. There are 2 board types of surveys:

 Attribute Surveys: Measure likes/ dislikes
 Research Surveys: Test hypotheses

There are 2 crucial aspects of surveys- development of a valid and reliable questionnaire and selection of the sample, which should be generalize the results to the population:

 Probability Samples: Sample selection depends on the sample frame
– Random Sampling
– Stratified Sampling

• Nonprobability Samples: Samples are not representative
– Quota Sampling- Samples with certain characteristics in the same ratio
– Convenience Sampling- Relies on participants who are accessible and available

However, to get the potential drawback, the researcher needs to ensure that he can get the respond to the questionnaire. A response rate of 70% or better should pose no threat of bias. For that reason, the important factors for doing surveys are Sample size and Final response percentage.

Moreover, the researcher should avoid the threats to internal validity in surveys that are:

 The questionnaires should be reliable and valid for the target population
 Any measuring instrument or scale used also should be reliable and valid
 Testing should be conducted under uniform conditions
 Administrators should be naïve with respect to any research hypothesis being tested
 Administrators should be trained and supervised
 To eliminate the possibility of a biased sample:
– Some steps should be taken to contact a sample of nonresponders to determine that they do not differ from the sample of responders

For that reason, it can conclude that the Caution Factors when evaluating surveys are:

 The sample frame
 The percentage of participants selected
 The criterion of exclusion
 The response return rate (less than 70%)
 The likelihood that characteristics of nonresponders introduced bias
 The nature of the questionnaire
 The reliability and validity of the questionaire
 The number and characteristics of interviewers
 The training and supervision of interviewers
 The uniformity of the interview setting

Conclusion

In this book, it presents 2 kinds of qualitative research that are Case Studies and Narrative Analysis.

For the Case Studies, they appear in all of the social sciences. They also can describe the behaviour by Observation, Interview and examination of documents.

For Narrative Analysis, observe group interactions by joining the group. It also includes a temporal sequence of experiences, some themes running through them and structural coherence.

However, the book also presents a quantitative method that is survey. It is conducted with the intent of obtaining a general sense of what people feel.

Even there are 2 types of method that are qualitative and quantitative methods, all of them include the same concept that is controlled observation, including Reliability and Validity. Thus, the researcher should test the reliability and avoid all of the threats to validity, including internal validity and statistical conclusion validity, to get the meaningful conclusion.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License