Comparison of Written and Oral Examinations in a Baccalaureate

Brigham Young University Brigham Young University

BYU ScholarsArchive BYU ScholarsArchive

Faculty Publications

2003-05-01

Comparison of Written and Oral Examinations in a Baccalaureate Comparison of Written and Oral Examinations in a Baccalaureate

Medical-Surgical Medical-Surgical

Patricia Rushton

[email protected]

Dennis L. Eggett

Follow this and additional works at: https://scholarsarchive.byu.edu/facpub

Part of the Nursing Commons

Original Publication Citation Original Publication Citation

Rushton, P. & Eggett, D. (23) Comparison of written and oral examinations in a baccalaureate

medical-surgical nursing course. Journal of Professional Nursing May/June, 19(3) P142-148.

BYU ScholarsArchive Citation BYU ScholarsArchive Citation

Rushton, Patricia and Eggett, Dennis L., "Comparison of Written and Oral Examinations in a Baccalaureate

Medical-Surgical" (2003).

Faculty Publications

. 495.

https://scholarsarchive.byu.edu/facpub/495

This Peer-Reviewed Article is brought to you for free and open access by BYU ScholarsArchive. It has been

accepted for inclusion in Faculty Publications by an authorized administrator of BYU ScholarsArchive. For more

information, please contact [email protected].

Comparison of Written and Oral Examinations in a Baccalaureate Medical-Surgical

Nursing Course

Patricia Rushton, RN Ph.D

Associate Professor

Brigham Young University College of Nursing

4015 So. 5200 W.

Salt Lake City, Utah 84120

Home Phone: 801-966-4897

Work Phone: 801-378-5375

Pager: 801-329-7425

Dennis Eggett, Ph.D

Director of the Center for Statistical Consultation and Collaborative Research

Department of Statistics

Brigham Young University

Comparison of Written and Oral Examinations in a Baccalaureate Medical-Surgical

Nursing Course

Abstract

Topic Investigated:

Nursing students have traditionally been evaluated with an objective written

examination. This method has demonstrated some benefits and disadvantages. This

project examined the value of oral examinations in evaluating nursing students.

Methods:

Five groups of students were evaluated with different forms of testing. Some

were evaluated with only written tests, others with only oral examinations, some with a

combination of both types of evaluations.

Results:

Results demonstrated that oral examinations can effectively evaluate the student’s

comprehension and application of clinical information in a clinical situation, as shown in

higher test results, compared with oral written examinations, and positive student

comments.

Conclusions:

Oral examination can be as effective or more effective in evaluating student

understanding of medical/surgical content and its application in clinical situations.

Key words: nursing student, examination, objective, oral

Comparison of Written and Oral Examinations in a Baccalaureate Medical-Surgical

Nursing Course

Introduction:

At a large private university college of nursing, students have been evaluated for

their comprehension of medical-surgical content with the traditional midterm/final

objective written test consisting of multiple choice, matching, and true/false items.

Continuing observation of this type of testing in one basic medical-surgical nursing

course has demonstrated some benefits and difficulties. Therefore, the course instructor

embarked on a project to determine the value of an oral examination, in comparison with

the traditional objective written examination.

Theoretical Framework:

The goal of nursing evaluation is to determine if the nursing student is able to

comprehend the information learned during study periods and taught during class work

and apply that information in the clinical situation. The hope is to develop evaluation

measures that will determine if such application is taking place. Bloom’s taxonomy, as

described by Slavin (2003), divides educational concepts into domains from simple to

complex. These domains can be used to describe test items and types of tests into the

ability of the test to actually evaluate student performance. The domains progress from

knowledge (recalling information) through comprehension (translating or interpreting

information), application (using principles or abstractions to solve novel or real life

situations), analysis (breaking down complex information or ideas into simpler parts to

understand how the parts relate or are organized), synthesis (creation of something that

did not exist before), and evaluation (judging something against a given standard). The

traditional written objective examination often fits into the "knowledge" category. The

hope for an evaluation measure would be to have it test at a level higher than knowledge,

requiring better understanding and application of the material in a specific situation.

Hypothesis, Purpose, and Goals:

The primary purpose of this study was to compare the effectiveness of traditional

objective written examinations and oral examinations in evaluating students’ knowledge

base in a basic medical/surgical nursing class. The study examined the effects of testing

style on student learning by comparing test scores and examination of comments on

student evaluations. The study hypothesized that oral examinations would result in:

1) More effective study habits and increased motivation to study information

from the course content.

2) Improved learning and knowledge base, critical thinking skills, and ability to

apply information in a specific clinical situation demonstrated by higher

examination scores

Background Research Studies and Significance:

Describing the use of oral examinations in any setting was limited in the

literature. Descriptions of their use in nursing settings were even more rare. However,

even though some of the following citations are older, they are felt to be helpful in

an examination of this area of study.

Balran and Farnsworth (1997) administered an oral examination twice a year in

their course, the Sociology of Health. Students were video taped as they orally answered

two questions. The grade was given after the instructor reviewed the videotape of the

student’s answer. Instructors noted that, though not statistically significant, the mean

scores for the oral examination were higher than the written examinations. Negative

aspects of the process were student anxiety and nervousness. In spite of considerable time

being given to prepare the students for this form of examination, instructors found the

administration of such a test physically exhausting for the instructors when more than six

students were scheduled consecutively.

Piane, Rydman, and Rubens (1996) studied learning styles of public health

students in predicting a preference for taking oral or written examinations. They found

learning styles did not predict preferences for oral or written examinations. The study,

however, did not evaluate the value of each type of test in actual learning.

Mandeville and Menchaca (1994) used an elaborate two-part system that tested

seven education students in a group environment. First, the students individually

presented a concept they understood and had prepared, and were critiqued by the group.

Second, students worked in two person teams. They were provided with a concept they

presented without previous preparation. A second student critiqued them. The second

student then presented another aspect of the same concept. The total examination took

108 minutes to administer to the entire group. Students were graded on a point system

with points being determined by a combination of scores from two instructors, the

members of the group, and the student being evaluated. The authors felt there were two

limitations to their examination: 1) the situation may not be efficient for a larger group of

students, 2) there may not be general acceptance of the form of examination as an

accurate measurement.

In an older study at a school of pharmacy in North Carolina, Lindley, Mackowiak,

Williams, and Hak (1986) instituted the use of oral examinations in a pharmacy course.

The student perceptions of such a testing method were favorable. Students felt the oral

testing situation stimulated learning in the test preparation period, was fair in its

administration, produced a manageable level of anxiety, and was a good learning

experience. They felt oral examinations enhanced the value of the involved course.

However, the impact of oral testing on the students’ test grades or final class grade was

not discussed.

According to Bashook (2000) oral examinations have been used to evaluate U.S.

physicians since 1917. He states that, “Currently, 15 member boards of the American

Board of Medical Specialties administer 17 different standardized oral examinations to

approximately 10,000 physician candidates annually.” Though variable scoring systems

are used, the examinations consist of standardized cases or questions administered by

examiners who are practiced in administering such examinations. Bashook notes, “The

standardized oral examination is one potential way to measure the clinical judgment of

professionals.” As an example of such oral examinations, Solomon, Reinhard, Bridgham,

Munger, and Starnaman (1990) describe the process of testing candidates in emergency

medicine. The examination consists of cases provided by the candidates from their own

practice in combination with cases provided by the examiner at the time of the

examination.

Older nursing literature examined the advantages and disadvantages of objective

written and oral examinations separately. Robbins (1975) noted that advantages of

objective written examinations included rapid evaluation of the students’ responses,

uniformity of measurement resulting from clearly defined terms, and ability to test fully

the information covered in the course. The biggest disadvantage was the need for

thorough validation of the test questions. Turner, Hillier, and Kershaw (1982) suggested

that objective question validation should 1) demonstrate that test items measure what they

are supposed to measure, 2) develop questions with a clearly defined answer, and 3)

determine the discriminatory power of a question. They noted, however, the validation,

though essential, is time consuming and monotonous. Marsden (1983) wondered if the

objective written test was even worth keeping in terms of its ability to actually evaluate

the ability to care for patients.

Comments on the value of oral or, at least, case oriented evaluations are available

in the literature. Marsden (1983) suggested an alternative to objective testing of patient

oriented case studies. He noted that such scenarios might indicate an understanding of

the medical condition of the patient and needed nursing care, and would test the same

knowledge content as that tested in objective testing. Dalis (1970) studied student

achievement in health education subjecting one group of students to precise behavioral

objectives and another to more vague behavior objectives. Results indicated learning was

improved with precise behavior objectives, and students in the experimental group asked

more questions during instructional sessions. These studies may lead to the notion that

learning and knowledge application is improved with precise scenarios requiring the

student to know and apply specific information.

At the college of nursing that participated in this study, students have traditionally

been evaluated for their comprehension of medical-surgical content with the

midterm/final objective written examination consisting of multiple choice, matching, and

true/false items. Continuing use of this type of examination has demonstrated some

benefits and difficulties.

Benefits:

1) Allows a broad sampling of knowledge in an objective fashion.

2) Can be easily scored and analyzed.

3) Can be easily administered to large groups with limited need for staff

monitoring.

Difficulties:

1) Can be very time consuming to construct items that actually test the large

amount of content. It may require an unmanageable number of items.

2) May represent highly subjective selections of questions. Content from some

areas may not be tested at all.

3) Entails difficulty in writing unambiguous questions.

4) Rewriting items to eliminate numbers one and two above and to protect test

security is very labor intensive.

5) Adequate studying is difficult, in light of the amount of content, but helping

students focus their studying through a test preparation session may require

extra meeting and study time.

6) Immediate feedback after the examination is difficult, sometimes impossible,

since there is not always time to do a post examination review.

7) Coordination in putting the examination together, working with a testing

center, and adjusting scores to reflect problems discovered during the

examination review or as a result of answer sheet scanning can be labor

intensive for faculty.

Johnson and Johnson (2002) noted some additional benefits and difficulties to

objective tests.

Benefits:

1) Assessment of knowledge quickly and efficiently

2) Prevent bias in scoring

3) Measure student knowledge without bias toward writing, grammatical, or

neatness skills.

Difficulties:

1) Rely on recognition and recall for assessment.

2) Require specific, predetermined answers

3) Penalize poor readers

With the above information, the course instructor felt that the case study method

of both presenting content and examining understanding and application might have the

advantages presented below. This method is more thoroughly described in the

“Methods” section of this paper.

1) As with the objective written examination, oral tests would maintain the goal

of encouraging students to learn and study by specific objectives

2) It should provide for consistent lecture/discussion/testing content from one

block and semester to the next, while still allowing for updating of material as

appropriate.

3) It would allow for evaluation of comprehensive course content in clinical

situations, thus providing a real world environment.

4) It would provide for immediate evaluation of whether the students understand

the information and can apply that information in a clinical situation.

5) It would eliminate the confusion and ambiguity that had been experienced

with multiple choice questions.

6) It may not add to the workload of the students for study or preparation, but in

fact, may decrease their workload by making clearer the content the students

are responsible to understand.

7) It would decrease faculty workload by eliminating the need to revise test

questions every semester, and by eliminating the need for pretest preparation

sessions and posttest reviews. It would also eliminate the faculty time used to

organize and coordinate testing.

8) It allows students to demonstrate their knowledge in more complex levels

according to Bloom’s taxonomy.

Description of Subjects:

This study viewed a collection of test scores in five groups of students enrolled in

the fifth medical-surgical course of the nursing program. These are outlined below.

Group 1- 55 students- The same group of students was given 3 written

midterms and one written final - Test average - 85

Group 2-150 students-1 written midterm and written final -Test average - 91

Group 3- 45 students-Oral examination only, no midterm-Test average - 99

Group 4- 92 students-1 written examination, 1 oral examination-Test average - 90

Group 5- 47 students- 1 written examination with questions resembling NCLEX

examination questions, 1 oral examination-Test average - 92

All the students who participated in the study were at the same level of their nursing

education and had the same type and amount of nursing student experience prior to

enrollment in the course involved in this project. Testing was mandatory and test

scores have been maintained as part of the instructor’s permanent records. Test scores

only have been presented in this paper, making the study participants anonymous.

Method or Procedure:

Students were divided into the following groups by convenience as changes to the

testing procedure progressed from objective written examination to various forms of oral

and/or objective written examinations combinations.

Objective Written Examination Only Group (Group 1 and 2)

Objective written examinations consisted of a midterm(s) and a final examination

of multiple choice, matching, definition, and short answer questions. Several instructors

contributed questions and graded their portions of the examination. Efforts were made

to design questions, which would test the students’ ability to recall information,

synthesize known information into separate concepts, and to apply information and

concepts to clinical nursing situations (critical thinking). However, there was not

time to validate questions in a non-testing arena prior to using them as an actual test

question. The examinations did not change between blocks (A block consists of six

weeks of instruction. Two blocks equal one semester) within the same semester group.

However, since one of the objectives of testing is to evaluate students in the fairest

manner possible, changes were made to improve the examination questions between

semester groups. These examinations of about 100 questions were taken in a testing

center, available over several days, and had no time limitations for the actual testing

period.

Oral Examination Only Group (Group 3)

A test was designed consisting of nine case studies containing 132 questions.

Questions were designed, again, to test the student’s ability to recall information

and to synthesize given information into separate concepts and to apply the

information and concepts in clinical nursing situations (critical thinking). During

the test design, a test key was written in which the significant content, concepts or

application involved in each question were defined. Correct content, concepts or

points of appropriate application were given a point value to allow the instructor to

quantify the correctness of the student answer. At the beginning of the block the

students were given the nine case studies and informed that it would be from these

questions an examination question would be chosen. The students were informed of the

examination design, and given six weeks to prepare. The examination was given in a

faculty office, and were administered and graded by the same instructor. Each student

chose one question from the 132 questions in the case studies out of a hat and had 10

minutes to respond. At the end of the 10 minutes, students were asked to determine the

number of points they felt they earned based on the test key. A discussion between the

student and instructor then followed to determine the final number of points given. This

examination was given at the end of the course, and there was no midterm. Because the

students had six weeks to prepare and were required to know the answers to all the

questions, the single question answered was worth the same number of points previously

given to both the midterm and the final, 250 points. It is interesting to note that students

in this class tended to deflate their grades when given the opportunity to evaluate

themselves.

Oral Examination and Objective Written Examination Group (Groups 4 and 5)

The teaching team in which the study course was involved decided to increase the

value of the examination portion of the final grade from 25% to 50% so final grades

would more reflective of examinations than written work. The course instructor for the

study group felt that it was a great deal of pressure on the student to increase the value of

one examination from 250 points to 500 points, (50% of the 1000 point grading system).

It was decided to have the student take a 250 point (125 question) objective written

examination and a 250 point oral examination. The same set of nine case studies

continued to be used as preparation for both the oral and objective written tests. This

format was used for 18 months. However, at the end of this period, the study course

instructor felt it was important to emphasize questions more consistent with the NCLEX

examination in the course’s written objective examination and efforts were made to

replace old examination questions with new examination questions that would be

consistent with this goal. Forty percent of the old examination questions were replaced

with new NCLEX-type questions.

Cases and questions for all study groups were written from the objectives and the

content areas presented in the course. All groups were given the benefit of a review

period prior to taking whatever form of final examination was given that group. This

content included pathophysiology and nursing care in the areas of oncology, hematology,

GI, reproductive, genitourinary, cardiology, respiratory, endocrine, and neurology.

Concept areas were also addressed. These included diagnostic and lab tests,

pharmacology, medical treatment, community and professional issues.

Statistical Analysis and Results:

A comparison of the study’s examination method groups demonstrated a

significant difference in the percentage of total possible points on examinations

4,385

=58.88, p<0.0001), as well as in the comparison of final grade scores on a four

point scale, A=4, B=3, etc (F

4,384

=5.85, p<0.0001). Though the groups differed in the

number of students, there were sufficient numbers in each group to obtain reliable

estimates. The analysis controlled for varying numbers because standard errors are

adjusted for sample sizes. The oral examination only group scored significantly higher

(Table 1) than the other groups on the four point scale. There was no significant

difference in the average grade for the other groups on this scale. This means that those

students who only took the oral examination performed significantly better than those

who took other types of examinations. The percentage scale showed no significant

difference between the written objective examination and oral examination combination

and the single mid-term with a final combination. All other groups were significantly

different from each other (See Table 1).

Student Evaluation Comments

Both positive and negative comments on student evaluations were recorded. The

negative comments were principally made about the written examinations. Students felt

that cases did not prepare them for the objective written examination. The material they

were expected to know was overwhelming, and a 125-question objective written test was

too long. Students commented that they would have liked to have more short written

examinations throughout the block instead of one long one at the end of the semester.

They felt that the single question oral examination at the end of the block was very

stressful. Except for the stressfulness of the oral examination, all of the comments about

the oral examination were positive. Students felt the cases helped them truly understand

the material and provided practice in applying the material to clinical practice.

Student evaluations did not reflect the change in study habits. However, the

principle investigator noted that the students formed study groups to research and share

answers to the questions on the case scenarios. The groups determined which members

would find answers to each of the nine cases and the groups gathered periodically to

share and discuss those answers.

Limitations of this Study:

Limitations of this study include:

1) The inability to validate the questions in the objective written examination

over time and with several student groups

2) The examination scores of the objective written examinations and the oral

examinations are difficult to compare accurately given their inherent

differences

3) The same instructor did all of the oral testing. Though that instructor

attempted to be completely objective in evaluating the student answers, it was

difficult.

4) The manipulation of testing strategies and changes in test questions after

the data collection began increases the difficulty of comparing data.

However, the changes in the tests were made in an effort to more

efficiently and fairly evaluate students’ ability to learn and apply

knowledge to a clinical nursing situation. Since this should be the main

objective for any student evaluation measure, the limitation in

comparison of study data is understood and accepted.

Discussion:

The initial hypothesis of this project was that oral examinations would result in:

1) More effective study habits and increased motivation to study information from

the course content.

2) Improved learning and knowledge base, critical thinking skills, and ability to

apply information in a specific clinical situation demonstrated by higher

examination scores

More effective study habits were seen in preparing for the oral examination.

Students not only studied individually, but also formed study groups with assignments to

find the answers to specific questions and then shared that information with other

members of the group. Though not stated as a specific goal of the project or anticipated

as an outcome, students used these study groups to promote collegial exchange of real

life situations by use of clinical case studies.

Improved learning, increased knowledge base, critical thinking, and application of

information to clinical situations were noted through significantly higher examination

scores with the oral examination than any other form of examination. Student comments

noted that they felt oral examinations were beneficial, in spite of the time it took to study

for the test and the stress the testing procedure produced. There was no comment from

students indicating that study for the oral examination took more time than studying for a

written examination.

The oral examination seemed to be very stressful for some students, perhaps

because the concept was new. Great effort was taken to discuss the test frequently with

students. Faculty was available to answer any student questions about the examination

and its content prior to the examination period. A review period was scheduled during

class time to review the case studies with students and answer any questions students had

not been able to answer on their own prior to the examination. In spite of these measures,

all students were clearly concerned about having to orally present information to a faculty

member in an examination situation, and two students were actually in tears when they

came to their final situation.

As mentioned above, it was impossible for the faculty administering the oral final

to be absolutely objective in grading the student. Having said that, however, it was

absolutely clear to the faculty whether or not the student knew the information. With rare

exception, students came prepared to answer the randomly chosen question in a complete

and thorough fashion.

Professional role and community content is very difficult to test objectively, and

has not been previously included in this course’s written medical/surgical objective

examination. In writing the case studies, however, content on community and

professional roles was included. There was some question, however, about actually

including the content in the examination period because it is often perceived as easier

information to provide orally. It was decided, however, that if the information was

important enough to teach and ask in the case studies, it was important enough to teach

and ask about in the examination situation.

As anticipated, oral examination was labor intensive for the faculty. The initial

writing of the case studies took a great amount of time to be sure that questions were

understandable and content objectives achieved. Since the case studies were not

confidential, there was no need to change questions every semester, and once written,

little revision was necessary. Since students were allowed, even encouraged to ask

questions about the case studies, answering these questions, often on an individual basis,

was time consuming. Finally, a review session was necessary prior to the actual

examination period to answer questions students had not been able to previously answer,

to review material students may have been unclear about, and to emotionally support

students in their ability to use an unfamiliar examination format. The actual examination

administration was very time consuming for the faculty. Since it was a research project,

one faculty member took responsibility of administering all of the oral examinations.

Even at only 10 minutes for each student, examination of 24 students took about 4 hours.

This is probably not more hours than it takes to grade any test with a subjective or

narrative component, however, no post test review was necessary since students already

had their questions clarified and received immediate feedback about their test results.

In regard to the objective written examinations, changes in examination questions

were frequently necessary, for several reasons. During the first years of this project,

some faculty changed their questions to reflect new material and maintain confidentiality

every semester. This was very time consuming and does not appear to have resulted in

better test scores. During the final part of the project, replacing previous questions with

NCLEX type questions was time consuming and labor intensive. However, it did

produce statistically significant improvement in test scores over any other type of

objective written examination or objective written / oral examination combination. This

may validate the fact that it is difficult for faculty to write and validate test items on a

frequent basis. Standardizing items over a period of time may be helpful.

The actual statistical analysis seems to indicate that there is no difference between

having one midterm, multiple midterms or only a final written examination in final

examination scores or final grades.

Final grades were influenced by the written work students completed, in addition

to test scores. Of 390 students in the study, 149 or 37% had final course grades not

different from their examination grades. Seven students, 1%, had lower final course

grades one grade lower than their examination scores because they did poorly on written

assignments. Sixty two percent (N=244) of the students had final course grades higher

(one to three grades higher) than their examination grades because their written work

raised their final grade. After the examinations was changed to be worth 50% of the total

grade, 86 students or 61% of that group (N=140) had grades that were higher than their

test scores. One missing grade was reported. This was due to a student receiving an

incomplete grade secondary to illness.

Conclusions:

• Oral examination can be as effective or more effective in evaluating student

understanding of medical/surgical content and its application in clinical situations.

• Oral examination is labor intensive for faculty, but appears to be no more labor

intensive for students than studying for the traditional objective written examination.

• Though oral examinations with a case scenario format are stressful for students, many

commented they felt they learned significantly from this format and it appeared to be

worth the stress.

• The standardization and validation of examination questions, as found in NCLEX

examinations, produces improved scores on examinations.

• Multiple examinations in any of the combinations explored in this study did not

produce better test scores. However, with a large enough sample of students, multiple

examinations might prove to produce higher NCLEX scores than with fewer

examinations.

• Final course grades may not correlate with examination grades if 50% or more of the

grade is made up of other course assignments.

REFERENCES

Anderson, J. G., Aydin, C.E., & Jay, S.J., (1994) Evaluating health care information

systems: methods and applications. Sage. Thousand Oaks.

Balran, A. & Farnsworth, B. (1997) Oral exams: an alternative evaluation method. Nurse

Educator 22 (4): 6-7

Bashook, P. (2002) Assessing clinical judgment using standardized oral examinations.

Dalis, G.T., (1970) Effect of precise objectives upon student achievement in health

education. The Journal of Experimental Education. 39:20-23

Johnson, D.W. & Johnson, R.T. (2002) Meaningful Assessment: A Manageable and

Cooperative Process. Boston. P.63.

Lindley, C.M., Mackowiak, J.I., Williams, D.M. & Hak, S.H. (1986) Ambulatory care

pharmacy practice course: introduction and evaluation of oral examinations.

American Journal of Pharmaceutical Education. 50:268-274

Mandeville, T.F. & Menchaca, V. (1994) Group oral exams: exploring assessment

techniques for new instructional paradigms. Reading Research and Instruction.

33(4): 319-325

Marsden, E. (1983) Nursing examinations. Nursing Times. Jul 6-12; 79(27):72

Piane, G., Rydman, R.J., & Rubens, A.J. (1996) Learning style preferences of public

health students. Journal of Medical Systems. 20(6): p.377-384.

Robbins, E. (1975) Examining examinations—2; objective tests. Nursing Times. Oct

23;71(43):1702

Slavin, R.E. (2003) Educational Psychology: Theory and Practice. 7

edition. Boston.

Pp. 466-467.

Solomon, D.J, Reinhart, M.A., Bridgham, R.G., Munger, B.S., &Starnaman, S. (1990)

An assessment of an oral examination format for evaluating clinical competence

in emergency medicine. Academic Medicine. 65(9): supplement p. 43-44

Turner, P.A.C, Hillier, V.P, & Kershaw, J.E.W. (1982) How objective are objective tests?

Some uses of the computer in validating multiple-choice questions. Nursing

Times, Supplement. Sept 22: quest 2-4.

TABLE 1

GROUP MEANS

Four Point

Scale for

Final

Grades

Percentage

of Total

Test Points

Scale

Group

Mean

Oral Only (group 3)

3.97

99.09

NCLEX/ Oral (group5)

3.87

91.98

Written and Oral (group 4)

3.83

90.06

Single Mid-Term with Final

(group 2)

150

3.82

150

89.81

Three Mid-Terms with Final

(group 1)

3.81

84.40

* Significantly different from all other groups at p<.05

# These two groups are not different from each other but are

significantly different from other groups at p<.05

Response to critique

1.Is it a significant and original contribution to knowledge in the field?

Question about reliability of statistics will be dealt with under question 2.

2. Is the experimental design, methods, and results presented, compete, and soundly

based?

It is unclear exactly what testing strategies were used in some areas. For example, it is

difficult to determine what was meant by “3 written midterms and a written final” for

Group 1. Did the same group of students take 3 examinations and a final or were there 3

groups taking one midterm and a final? See bolded print on page 11.

Groups consisted of varying numbers of subjects raising questions regarding the

uniformity of groups and the strength of statistical analysis. The uniformity of the group

is dealt with on page 12, paragraph 1. The statistical analysis is clarified on page 15

under the “Statistical Analysis” section.

There was manipulation of testing strategies and “frequent changes in test questions”

after the data collection began leading to questions regarding the reliability of the data

obtained for groups 1-4. This limitation is more fully explained on page 17.

It is unclear as to what specific criteria were used to make judgment regarding scoring

for the “oral examination” group. Please note changes on page 13 under the oral

examination group.

Testing of critical thinking was not defined. See bolded area on page 12.

Final grades used in the analysis, represented scores from course projects as well as

scores from the various testing groups measured. This leads to significant questions

regarding the differences in testing strategies along. This portion of the “Results”

section has been rewritten to clarify the fact that testing scores alone were compared

separate from the final grades. See page 15.

3. Do the author’s conclusions logically flow from the data?

The conclusions may be affected by the reliability of the data as cited above. Data

reliability has been dealt with under question 2.

4. Are the references pertinent, current, and in APA format?

Several articles cited are dated. The reason for this is clarified on page 5. The reference

to Alverno College has been deleted.

5. Are illustrations and tables adequate in quality and number?

Table 2 does not differentiate between group differences. Table 2 has been omitted.