RUNNING HEAD: Determining Success
Determining Success Potential in School
Curt Oechsle
Determining Success Potential in School
Introduction
There are several ways that teachers, school administration, and other educational institutions assess a student’s success. Often, the basis of their system of measurement is found within standardized tests and other assessment activities or examinations. The assessor then uses the information gathered to determine a student’s success based on percentages and grade point averages (GPA). The question that arises is whether or not the tests and other examinations (written, orally presented, etc.) are fair representation of a student’s success. With this question in mind, I developed two diagnostic systems to determine a student’s success as opposed to grades and other measurement devices currently in place.
Diagnostic System 1: Pressure and Support
The first diagnostic system is in place to determine the level of support and level of pressure the student has from teachers, family, peers, etc. It is hypothesized that if a student had relatively strong support and strong to low pressure on them to succeed, then they would most likely do so. On the other hand, if a student has low support from the people around him or her, and strong to low pressure on them to succeed, they will be less likely to do so. Therefore, in the first diagnostic system, four mutually exclusive categories were created to attempt to capture the level of support and pressure of the individual who is being diagnosed. The four categories were: Strong Support/Strong Pressure, Strong Support/Low Pressure, Low Support/Strong Pressure, Low Support/Low Pressure. The support is measured by observing how much attention is placed upon the student, and the pressure is measured by observing the pressure applied by the people around the student.
Although the system may seem simple, it also has its complexities. The problem that exists with the first system is that it is subject to being observer biased. The support and pressure of the student is left for the diagnostician to determine, not necessarily the system itself as perhaps the grading system that is presently in schools may be able to do.
The grading scale system bases itself in the number of questions a student answers correctly and incorrectly, and the system will strictly reflect the assessment of the student without a need for an observer to determine the correctness or incorrectness of an answer. This is not always true of the grading scale, because of activities such as written assignments and other such examinations of knowledge. My first diagnostic system aims to adequately measure the effects of the environment that a student is surrounded by, whether that is positive or negative. The information that is lost, however, is that the diagnostician cannot see the effect of the support and pressure of the student. The student may have strong support and pressure on them to succeed, but they may not participate in class or other activities. Thus, important information is lost, even though some valuable information is gained about the student’s surroundings and social learning environment.
Diagnostic System 2: Level of Participation
The second diagnostic system was created to measure the level of participation a student subjects themselves to within the classroom. It is hypothesized that if a student is highly involved in a classroom, the likelihood of their being a successful student would be greatly enhanced. The opposite side to this continuum is that if a student is fairly reserved and elects to withdraw from participation within the classroom, the likelihood of their being a successful student would be prone to decrease. Three mutually exclusive categories were created to adequately capture the level of participation a student has within his or her classroom. The three categories are: Highly Involved, Moderately Involved, and Minimally Involved. To be considered as a “Highly Involved” student, the person being assessed has to actively participate in nearly all classroom activities. If a student is acting at this level within the classroom, they would be more likely to have their mind engaged in what was going on in the classroom and would most likely learn better as a result. To be considered as a “Moderately Involved” student, the student had to participate actively in some of the classroom activities, but not all. This would most likely describe the “average” student. To be considered as a “Minimally Involved” student, the person being assessed must rarely participate in any of the classroom activities. This person would most likely be less engaged in what was going on in the classroom and may not learn as much as they could if they were more engaged in the classroom. The term, “activities” is being used to describe classroom discussions, group work, and other similar activities.
There are also some advantages and disadvantages to this diagnostic system as well. The advantage to this system is that the diagnosticians can clearly see how a student functions within the classroom. This can give the assessor a decent picture of how hard a student attempts to learn new things. However, the information lost is just as important as the information gained. In this diagnostic system, the diagnostician loses information on the social learning environment of the student. If the student has low support from the people whom surround him or her, then the student may have a low desire to learn and to become actively involved in the class. There is also no way of knowing if the student understands the material he or she actively participates in. Just because they participate in class does not necessarily mean that they understand the material that is being presented to them.
Method
Participants
Ten diagnosticians from an abnormal psychology class at a Midwestern university were designated at random as the participants for this assessment. There was also one Resident Assistant from dorm hall at the same Midwestern university to be designated as the expert. He is a fifth-year senior double-majoring in social studies education and Christian ministries.
Materials
The ten participants that were designated as the diagnosticians as well as the Resident Assistant were given brief summaries of three case studies (See Appendix A-C) and three diagnosis papers for each case study (See Appendix D). The diagnostic sheet also had a section available for any additional comments that the diagnosticians may have had.
Procedure
The professor of the abnormal psychology class handed out the case studies and diagnostic sheets to ten students in his class at random. The estimated reliability for each diagnostic system was 70%. The reason I chose 70% as an estimate of reliability is based upon a possible complexity distinguishing among the different categories as well as any complications within the case studies themselves. The estimated validity of each system was 80%. I anticipated that the expert would most likely make a diagnosis that was extremely close to the diagnosis of the majority of the other diagnosticians. I also anticipated that the expert would have a more reasoned understanding of children and their classroom situations. The reliability of each system will be determined by comparing the diagnoses of the 10 diagnosticians. This will be done by adding the most recurrent diagnosis for each case and dividing that number by the total number of diagnoses to receive a percentage of reliability for each system (See Figure 1). The validity of each system will be determined by weighing the diagnoses of the 10 diagnosticians against the designated expert’s opinion and receiving a percentage for each system (See Figure 1).
Results
Diagnostic System 1 was found to be 46.7% reliable and 43.3% valid, and Diagnostic System 2 was found to be 60% reliable and 60% valid. See Figure 1 below.
|
|
|
|
|
|
|
|
|
Cases |
|
Decision |
SALLY |
BOB |
BRIAN |
|
|
|
|
|
|
|
|
|
|
System One |
|
Strong/Strong |
0 |
7-X |
2 |
|
|
|
|
Strong/Low |
4 |
2 |
2 |
|
|
Reliability=14/30=46.7% |
|
Low/Strong |
3-X |
0 |
3-X |
|
|
Validity=13/30=43.3% |
|
Low/Low |
3 |
1 |
3 |
|
|
|
|
|
|
|
|
|
|
System Two |
|
Highly Involved |
0 |
8-X |
2 |
|
|
|
|
Moderately Involved |
4 |
1 |
4-X |
|
|
Reliability=18/30=60% |
|
Minimally Involved |
6-X |
1 |
4 |
|
|
Validity=18/30=60% |
|
Reliability à Validity à |
10/20=50% 9/20=45% |
15/20=75% 15/20=75% |
7/20=35% 7/20=35% |
|
|
|
|
|
|
|
|
|
Figure 1. A graph showing the diagnosis of the diagnosticians for each of the three cases presented to them. The “X” beside certain numbers indicates the expert’s diagnosis of the three cases.
Discussion
The results were far from the prediction of 70% reliability and 80% validity prior to the examination of the diagnoses. I believe that the discrepancy for the large margin of error can be explained by a few factors. The first was an error I made in the actual diagnostic paper that may have affected the results in one way or another. At the very least, it most likely caused some confusion. A second factor would be a lack of explanation for the mutually exclusive categories so that the diagnosticians could more accurately place the case studies in the appropriate category. A third possible factor could have been a lack of clarity within the case study itself. Perhaps all of the details that were necessary in diagnosing a person into a category were not crystal clear or simply obvious.
In the diagnostic paper that the diagnosticians used, as well as the expert, there was a grammatical error. In the “Low/Low” category, the description of that category was identical to the description of the “Low/High” category and has since been fixed. However, this may have had some effect on the decision made by the diagnosticians. Another theory that came to mind for the low reliability and validity numbers may have been a result of lack of time to thoroughly complete the diagnosis for each diagnostician. During the class session, each diagnostician had approximately 50 minutes to read through my three case studies and complete their diagnosis of them. However, in that same 50 minutes, they also had several other case studies and diagnostic systems to complete from their other peers as well. It may have resulted in a quick decision as opposed to a thought-out decision.
There are several things that I have learned from this diagnostic test. Although I think that the reasons for all the discrepancies that I listed above are quite valid reasons, I believe that the full weight of understanding what it takes to create a near flawless diagnostic system has been brought upon me. At the beginning of the semester, I saw this project in the future and it seemed disastrous. To put it simply, I was afraid of it. However, as time rolled on, and after I heard what was required, my mind changed. I thought it would be more or less a piece of cake. I chuckle at that thought, but have now come to the realization of what is necessary to creating a system that is helpful in many ways. Perhaps the greatest amount of knowledge that I have gained is that when I created the diagnostic systems, I made them in such a way that I would be able to understand what they mean and what I mean by them. However, the problem with that is that I am not everyone else. I had to think through the eyes of others and try to describe my diagnostic system in a way that most people would be able to understand, as well as make it capable for them to see what I was getting at. This was easier said than done. Although I had revised my diagnostic system, it still didn’t measure up to where I expected it to.
Validity and reliability are also something that I was trying to learn and wrap my brain around. I think this particular assignment was terrific for me to fully understand what the meaning of validity and reliability are. It seems so much clearer and easier to understand when you can tangibly apply them on your own. It is also hard to make your diagnostic system reliable and valid from random case studies that apply to partial tidbits of your total diagnostic system. My eyes were given yet another larger breadth of view in the realm of my education. It is incredible to note how individualistic even minute problems can be. They can mean something completely different from one person to the next.
I really appreciated this assignment and I firmly believe that this gave me a much better understanding of validity and reliability. Prior to this class, validity and reliability had been merely terms from the statistics class that I had once taken. It makes a difference with the context you learn them in. I am truly glad that I can now apply them and use them in the future.
Appendix A