Evaluation Research Articles
Empirical Research
The based of empirical research should include controlled observation, which are Reliability and Validity. For the Controlled observation, the researcher should:
• Control the conditions when data are collected
Noise is eliminated/ minimized
Gathering data under uniform conditions (“Superfluous factors”)
Every participant has the same condition (“Environmental distraction”)
These features ensure that the collected data will be reliability and validity. For Reliability, it can refer that the researcher should:
Under that same experimental conditions anyone should be able to obtain the same results
Also refers to precision of our measuring instruments
Data Scores include two parts that are Systematic variance and error variance as the below formula.
Data Scores = Systematic variance + nonsystematic or error variance
There are several reliability test procedures that are:
Test-Retest: The same group of people is given the test on 2 occasion- The scores will be the same in each time
Split-half: Compare the 1st half vs 2nd half- if all items measure the same characteristic, there should be a relationship or correlation between the scores on each half of test
Cronbach’s alpha: measures of reliability are in the form of correlation coefficient that ranges from 0.00 to .99 (at least .65)
Besides the reliability, there is another factor that is Validity. For the validity, it refers to the conclusions are valid, meaningful, and useful. There are several Validity Testing procedures that are:
• Content Validity: refers to the extent to which all items reflect the characteristic
• Criterion Validity: Split participant as the test scores that correlate with a behaviour
• Predictive Validity: Predict behaviour or performance as test score
• Construct Validity: The characteristic cannot be measured directly (Construct Validity is the most difficult to establish)
However, to ensure that the conclusions are valid, the researcher needs to avoid the threats of Validity. Validity can divide into 2 types that are Internal Validity and Statistical Conclusion Validity.
For Internal Validity, there are 2 types of the qualitative studies that are “Studies with pre- and posttests” and “Studies with or without pre-and posttests”. Each study gets different threats as the following:
Studies with pretests and posttests
History: Directly or indirectly could affect the behaviour
Maturation: Refers to any changes within the participant that occurs during the test
Instrumentation: Refers to any changes in the measuring instrument from pre- to posttest
Initial Testing: A change in posttest performance that results from pretest experience
Regression toward the mean: When pretest scores were extremely high or low, posttest scores are predicted to be less extreme, regardless of treatment effects
Studies with or without pre- and posttests
Selection bias: Difference in performance may be associated with a participant characteristic
Selective loss: Loss of particular participants from a group
Diffusion of treatment: The unintentional spread of treatment to a control group
Compensatory rivalry: Participants receive unequal treatment
Resentful Demoralization: Participants resent the lack of experimental treatment
Experimenter expectancy: Participants behave as the hypotheses
For Statistical Conclusion Validity, there are 2 types of well-known errors that are called “Type I” and “Type II” errors.
• Type I: Reject the Null Hypothesis when it is true
• Type II: Retain the Null Hypothesis when it is not true
False True
Reject H0 OK Type I
Accept H0 Type II OK
There are a lot of reasons that cause “Type I” and “Type II” errors that are:
Insufficient power of statistical test: : If samples are very large, any effect can be significant, and this could lead to a type I error
Unreliable instrument: If the measuring instrument is not reliable, performance will be variable
Varied test conditions: If testing conditions are not uniform, performances will be variable
Varied participant characteristics: If participants have different characteristics, performances will be variable
Violation of statistical assumption: If at least one is seriously violently, the analyses may fail to reveal a difference
Fishing: When an unreasonable number of statistical tests are conducted on the same data, one may reveal what appears to be a significant difference
For the Empirical research, the reader should keep understand the article step by step as the following:
Rationale: The past literature aroused the investigator’s interest in the subject
Purpose: let you know what exactly the investigators intended to accomplish. Often, it is expressed in terms of hypothesis testing
Method: illustrate how they were recruited
General Procedure: what was done to the participants or what they were required to do
Results: Appropriate analyses of the data
Discussion: Conclusion regarding the outcome of their manipulation
Case studies
Case Studies appear in all of the social sciences and are examples of qualitative research. They range from those that are purely descriptive, explaining behaviour. In all cases, data can gathered by Observation, Interview and examination of documents. However, there are the Caution factors in Case Studies that are:
The reason the case was selected
The nature of intervention
The objectivity of behavioural assessment
The assessor of behaviour
The past and/or present factors that could change the behaviour
The method of selecting participants in a singe unit
The reliability and validity of interview questions
Investigator bias
Narrative Analysis
Many qualitative methods exist besides the case study. Narrative Analysis is one of them. The researcher will observe group interactions by joining the group. It also includes a temporal sequence of experiences, some themes running through them and structural coherence. In this analysis, most of the interview questions are generally open-ended.
There are 5 aspects to a narrative report of one’s experiences of an event:
Attention to certain phenomena make them meaningful
Narration is the telling of the event in the form of a story
How the story is told depends on the listener- narratives about the same event will differ when the listeners are friends, colleagues or oversees
Transcribers typically order narrative's experiences in sequence
Analysis of the transcribed tape- involves decision about what will be included and excluded according to the researcher’s anticipated responses
However, different readers will attribute different meaning to the text depending on their personal experiences and expectations.
The core of the analysis is Trustworthiness of the interpretation, which will depend on:
The nature of the questions asked in the interview
The place in which the interview occurred
A different type of analysis may lead to a different interpretation
For that reason, the Caution Factors in Narrative Analysis are:
The reason the particular individual was interviewed
The location in which the interview took place
The nature of the interview questions
The relationship between the interviewer and narrator
Survey
Surveys are almost always quantitative method to the population of interest. They are conducted with the intent of obtaining a general sense of what people feel. There are 2 board types of surveys:
Attribute Surveys: Measure likes/ dislikes
Research Surveys: Test hypotheses
There are 2 crucial aspects of surveys- development of a valid and reliable questionnaire and selection of the sample, which should be generalize the results to the population:
Probability Samples: Sample selection depends on the sample frame
– Random Sampling
– Stratified Sampling
• Nonprobability Samples: Samples are not representative
– Quota Sampling- Samples with certain characteristics in the same ratio
– Convenience Sampling- Relies on participants who are accessible and available
However, to get the potential drawback, the researcher needs to ensure that he can get the respond to the questionnaire. A response rate of 70% or better should pose no threat of bias. For that reason, the important factors for doing surveys are Sample size and Final response percentage.
Moreover, the researcher should avoid the threats to internal validity in surveys that are:
The questionnaires should be reliable and valid for the target population
Any measuring instrument or scale used also should be reliable and valid
Testing should be conducted under uniform conditions
Administrators should be naïve with respect to any research hypothesis being tested
Administrators should be trained and supervised
To eliminate the possibility of a biased sample:
– Some steps should be taken to contact a sample of nonresponders to determine that they do not differ from the sample of responders
For that reason, it can conclude that the Caution Factors when evaluating surveys are:
The sample frame
The percentage of participants selected
The criterion of exclusion
The response return rate (less than 70%)
The likelihood that characteristics of nonresponders introduced bias
The nature of the questionnaire
The reliability and validity of the questionaire
The number and characteristics of interviewers
The training and supervision of interviewers
The uniformity of the interview setting
Conclusion
In this book, it presents 2 kinds of qualitative research that are Case Studies and Narrative Analysis.
For the Case Studies, they appear in all of the social sciences. They also can describe the behaviour by Observation, Interview and examination of documents.
For Narrative Analysis, observe group interactions by joining the group. It also includes a temporal sequence of experiences, some themes running through them and structural coherence.
However, the book also presents a quantitative method that is survey. It is conducted with the intent of obtaining a general sense of what people feel.
Even there are 2 types of method that are qualitative and quantitative methods, all of them include the same concept that is controlled observation, including Reliability and Validity. Thus, the researcher should test the reliability and avoid all of the threats to validity, including internal validity and statistical conclusion validity, to get the meaningful conclusion.