Textbook
for the course:
Educational
Testing & Measurement
ISBN:
978-0-470-52281-3
NOTES:
Thursday, May 16, 2012
The Progression is an inverted pyramid: Educational goal on top, general objective next, then IO last.
Educational Goal (Global Objective)
Broad statements of what students will be able to do. Framed within a relatively broad content domain. Tend to be lofty or noble. Take a very long time to actually accomplish. May not be in one day, could be in a lifetime. These are usually not directly measurable. Provide a good starting point, but not specific enough to be an IO.
General Objective
More specific than educational goals. Framed in terms of student behavior. Tied to actual classroom instruction. Usually called “state standards.” National standards? Kept switching back and forth. Still too broad to make good IOs. Useful for curriculum design and planning large instructional units. Generally, where students by the end of the unit.
Structure of State Standards:
1) Performance Standards (proficiency levels)
a. Measure of how well student is able to demonstrate the learning.
b. Describes the level of performance that students should achieve.
c. NCLB calls these academic achievement standards.
d. AZ Dept. of Ed. ex: falls far below, approaches, meets, exceeds (FAME).
2) Content Standards
a. General statement that describes what students should know/skills they should be able to do.
b. NCLB call these academic content standards.
c. Usually pretty general.
d. AZDE ex: Math, Grade 2, Strand 1, Concept 1: Understand and apply numbers, ways of representing numbers, and the relationships among numbers and different numbers systems.
e. English, Language Arts and Math.
f. Organized by:
i. Domain
1. Groups of related standards and clusters
2. Fifth (5th) grade math example: # and operations in Base Ten (10) – 5.NBT
3. Cluster (NCLB calls these content standards)
a. Groups of related standards
b. Ex: Understand the place value system.
4. Standard (NCLB = benchmarks)
a. Ex: 4. Use place value understanding to round decimals to any place.
b. Abbreviated for this class 5.NBT.4
i. Consecutively numbered no matter how many clusters.
3) Benchmarks – “the demonstration”
a. Specific skills within the broad content standards.
b. Students must do these to demonstrate they met the standard.
c. Specifies what the student is to actually perform.
d. Similar to IOs, but not designed specifically for classroom use.
e. AZ calls these performance objectives (PO).
i. AZDE ex: Math, Grade 2, Strand 1, Concept 1, PO 1. Express whole numbers 0 to 1,000 in groups of hundreds, tens and ones using and connecting multiple representations.
Points to Clarify
1) Every state has to have these three (3) levels, but the terms vary by state.
a. E.g. Some states use the term “proficiency levels” for “performance standards” and use “performance standards” for benchmarks.
b. Content standards do not mandate what happens in the classroom.
i. They are what is to be learned NOT a way of learning!
ii. They provide a broad framework for curriculum.
c. Districts often provide their own specific benchmarks as well.
i. Specific to district needs.
ii. Assessed using a benchmark test.
ABCD’s of objectives
A) Audience: who is doing the behavior?
a. Instructional objectives are about the students.
B) Behavior: what is to be done?
a. How the behavior is demonstrated.
C) Conditions: what are the constraints or allowances?
a. Permitted or not materials (calculators, dictionaries), time to complete, alone or in groups.
D) Degree: how well?
a. Determination of success or failure, moderate levels of success.
E) Everyone: SHOULD AT LEAST CONSIDERED WITH ALL
‘OBJECTIVES.’
To Start
1) Look at the standards in your area.
2) Create a content outline for the unit.
a. Components of the unit in outline form
3) Think about what a substitute about what a substitute teacher would need
a. Even if you plan to be there throughout the unit
b. It’s a good practice for keeping it clear – remember that others may ask to see these! i.e. Parents, administrators.
c. Focus on the students and their performance – NEVER on the teacher.
d. Focus on the learning product …
Writing
1) Beginners should use a general stem (“SWBAT” – Students will be able to …)
i. Reminds you of who the audience of these objeves are.
ii. Feel free to omit general stems once you aree comfortable with the format.
b. Focus on observable and measureable behavior (use good action verbs).
c. Include examples if using general or vague terms.
d. Write a few at different Bloom’s levels and determine which are best.
i. Remember that higher-level skills include lower-level skills.
Don’t Do:
1) Use double-barreled objectives (i.e. an objective that includes two
goals/objectives in one ‘objective’ in benchmark objectives.
2) Get too specific or general
3) Include objectives that seem like they will be difficult to teach or
measure.
a. These are education goals.
b. Fine to have, just don’t call them IOs.
What do you think?
1) Students will learn how to read technical papers. (Should be: Students will be able to identify common parts of a technical paper).
2) Students will be able to read the following paragraph out loud with fewer than four (4) errors. (TOO SPECIFIC, IT’S A SPECIFIC TASK)
3) Students will be able to identify at least four (4) of the major battles of Civil War and identify the causes of the Civil War. (DOUBLE-BARELLED)
Formal vs. Informal Assessment
1) Formal: usually planned in advance.
a. Systematic (e.g. design issues/grading criteria)
b. Relatively infrequent.
2) Informal: less structured
a. Relatively frequent.
b. Typically not graded.
c. Common strategy – asking questions
i. Consideration for question asking
1. Stay on content
2. Avoid generic or vague questions
3. Take environment into account (Year in school/Grade level)
4. Be aware of your patterns (bias) in calling on students.
a. Both male and female instructors call on boys more often than girls.
b. Action Zone: Group that interacts with the teacher, one that does.
5. Be aware of wait time
a. Low level Questions: 3-5 seconds
i. Adopting a required wait time may produce more student participation.
b. High level Questions: 10-30 seconds
6. Reword questions if students seem confused.
7. Use follow-up questions.
a. To allow others to answer or to get students to delve deeper on a topic.
8. Be careful how you respond to incorrect answers.
a. Mediate. “That’s not how we use it in this context.”
Tuesday, May 15, 2012
Analyzing
standardized test…
1) Cut-offs
change from year-to-year
2) Broken
down by: Strands & concepts
3) Norm-reference
test (NRT) [i.e. Stanford 10]
•
Scores are 1-9. Ranges are 1-3, 4-6, 7-9
Arizona Interment for Measurement Standards
(AIMS) was not required for 2011-2012 cohorts to graduate high school. Could
possibly be required for 2012-13 cohort to graduate from high school.
High-Stakes Testing (HST) proponents: ensures all students are held to the
same standards and that everyone takes learning more seriously.
Gives an =
way to measure for making comparisons and determining accountability.
HST is another
tool in the assessment toolbox to assess.
Some
research says it allows learning to transfer to other measures and/or areas.
2004 polls
(No Child Left Behind – approved in 2002, enacted 2004): > 70% support some
HST practices.
HST used as part of educational reform and accountability.
Most cite need for objective measures.
HST critics:
Emphasizes
rote learning!
Too much
time is spent on preparing students of the test rather than on meaningful
learning.
May not be
the most appropriate way to measure learning. Don’t measure creativity, freedom
of expression.
May penalize
teachers who work with academically struggling students.
Some
indications suggest that HST has cultural bias.
HST is often
a misused tool.
Increased
cheating can happen with HST. Scandal in Georgia (1-2 years ago, teachers,
administrators were fired.)
Some
research correlates (no correlation) HST with increased dropout rates.
HST scores correlated
with decreased property (poverty in school districts) values.
American
Educational Resource/Research Association’s (AERA) criteria for HST
No single test
should be used for decision-making.
Provide
adequate resources and opportunity to learn the material before test.
‘Validation’ for intended use to measure
‘intelligence’ and there are
sufficient indicators that are reliability.
HST ‘should’ provide full disclosure of
potential negative consequences.
The test and
the curriculum ‘should’ be aligned.
‘Validation’ of passing scores and
achievements for schools and pupils are an important result of HST.
Meaningful
remediation courses/course work for failing examines (not as many or as
intensive as in the past, because the budget for these programs was cut).
Attention on
HST is given to those with language differences.
Attention to
individuals with learning disabilities. (Varies from district-to-district.)
Five years
ago a school allowed learning-disabled students to use calculators on the math
portion because of their Individual Educational Plan (IEP).
Adherence to
testing rules for whom will be tested.
Teaching to the Test?
Often gets misinterpreted as “teaching the test.”
Often gets misinterpreted as “teaching the test.”
Instruction
is driven by the test’s content domain.
So, the test
is not driving your instruction, your objectives are:
Teaching
general test skills.
Focused
on learning processes/life-skills.
Teaching the test
Instruction
is driven by what items will be on the test, and in some cases the direct
content that will be on the exam.
A test
shapes how teachers instruct pupils.
Teaching
revolves around the test.
Teachers who
have to ensure their students’ success on HST should ‘Teach to the Test,’ but
most ‘Teach the Test.’
Improper Prep Techniques
Don’t give
them exact test they’ll be taking.
Don’t give
them a parallel form of the test they’ll be taking.
Don’t find
out the answers to the items on the test and give them to your students.
Don’t
deliberately deceive your students about the directions, intent, or content of
the tests.
Proper Prep. Tech.
Evaluate
your own biases and be objective about the need for preparing.
Be
knowledgeable about the test, purposes, and scoring.
Familiarize
students with the format of the HST.
Practice
with released sample items.
Prepare them
for the length of time they will be testing. Tell students to bring
books/activities to do if they finish the test early.
Don’t
test for four (4) days straight, but give them practice testing for 60 or 90
minutes at a time.
Many
teachers allow, food, free time after testing to break it up.
Teach
general test skills.
Teaching Test Skills
Definition:
Skills that allow students to more efficient and effective test-takers.
Suggest that
students:
Jot down any mnemonics/reminders before taking the test, if
there is free space.
Skim entire test.
Answer any you know immediately.
There may be cues/reminders in later items to assist with
answering earlier items.
Make note of item weights. (1 pt. item vs. 10 pt. item, focus
on 10 pt.)
Make a mark on items that you need to think about and come
back to.
With multiple choice, eliminate response choices that you
know are incorrect.
If there is a penalty for guessing, you should try anyway if
you can eliminate even
a single distractor. Can lose extra points if the test ‘graders’ if they determine if a
student is ‘guessing.’
Gender distinction: Men are more likely to guess on a test
question and they often get higher scores for ‘guessing.’
With true/false, underline or make note of parts that make
the item false.
Use poor test creation to your advantage – look for
grammatical or other parallels
between stems and response choices.
Review the test before turning it in.
Do NOT change your first answer
unless you completely misread it the first time. Subconscious forces typically
direct you to the correct response.
Make sure you’ve answered everything.
Sit next to smart kid (JOKE!)
Teacher said to use the same pencil they take notes with
because it has all the answers.
Bring candy to test, possibly.
Sit in the same spot you do in class when taking a test.
Keep a similar routine! Go to the bar, if you go to the bar
before a test. (LOL)
Know reliability vs. reliability for
test
Creating & Measuring Objectives
(Ch. 6)
-
Explain
the purose of Bloom’s Taxonomy and why it is impt.
-
Planning
assessment
-
Instruction
Objectives (I.O.)
Create well-written
I.O.’s for use in classrooms.
Bloom’s Taxonomies (1954)
Bloom is one
of the most significant influences on modern education. Author of “one of the
most widely cited yet least read books in American education.”
What’s
taxonomy: a way of classifying something hierarchically (low to high)
Bloom
organized behavior
-
psychomotor
-
affective
-
cognitive
– this is where we focus most
Pyschomotor
Present at
birth, uncontrolled, unlearned responses.
Basic-fundamental
movements: More skilled, intentional movements.
Perceptual abilities:
sensory information, sorting through stimuli.
Physical
abilities: involve flexibility and strength.
Skilled
movements: complex movements toward a complex goal.
Non-discursive
communication: non-verbal communications.
Use of psychomotor taxonomy
Not as
useful for most of our purposes.
More
informative for those working with children who:
Are
younger
Are
in physical education
Have
physical difficulties
May create
objectives related to classroom behavior that stem from psychomotor.
Teachers
should watch for difficulties in any of these areas.
Bloom’s affective taxonomy
Hierarchy of
methods to measure feelings or motivations.
Attention (receptiveness): aware, willing, and selective
perception of the transmission of information.
Responsiveness: active, willing, satisfying (to the
student), and participation.
Valuation: acceptance, preference, and
commitment expressed.
Organization: recognition or the value system and
prioritization among values.
Behavior according to values: avoidance of hypocrisy, general or
within identified limits.
Use of
Affective taxonomy
The hope is
that students value learning and internalize these values.
Can
help students see the inherent benefit to the material. Help them in life, not
just with academia.
Encourage alignment of behavior and beliefs.
Move beyond external motivators (grades).
Be able to identify and defend their belief systems.
A lot of
crossover of Bloom’s taxonomy, it can be cognitive; but more strongly is
identified in an affective manner.
Bloom’s Cognitive Domain
Use Bloom’s
taxonomy to organize instructional objectives.
Ideal
teaching should focus on higher-level cognitive skills.
But
learning at higher levels depends on prerequisites knowledge.
Prerequisites
knowledge is the result of having attained lower-level objectives.
Lower-level
cognitive skills
Knowledge
Using
recall and recognition
Accessing
rote memory of facts, terms, basic concepts, sequences,
classifications, principles,
theories.
Comprehension
Grasp
meaning/understanding
Demonstrated
by organizing, comparing, interpreting, describing, and
summarizing main concepts.
Higher-level
cognitive skills
Application
Using
knowledge in novel situations.
Requires
applying knowledge in different ways.
Analysis
Breaking
down a complex idea into parts.
Understanding
relationships and causes.
Making
inferences and identifying support for those inferences
Synthesis
(2nd to the most complex)
Putting
together several ideas to form new patterns.
Finding
new patterns or alternative solutions.
Often
demonstrated by production of a new product.
Evaluation
Evaluating
and assessing complex ideas
Presenting
and defending judgments and positions
Mnemonic
Devices
How
can you remember the order: Knowledge, Comprehension, Application,
Analysis, Synthesis, Evaluation?
One
way: Kids Can Always Achieve Standards-based Education
Revised Bloom’s Taxonomy (2001)
Bloom (1954)
was interested in classifying levels of behavior.
New taxonomy
was written for a different audience and purpose (K-12 educators).
Increased
focus on categorizing objectives and then aligning activities and
assessments with those objectives.
Increased
use of subcategories.
Tendency
to reframe in terms of verbs – learning is action-oriented.
Two New Dimensions
Knowledge
becomes it own dimension à remembering goes into cognitive
domain.
Cognitive
domain revised to contain other six (6) levels: evaluating comes before
Creating (Synthesis) now. Critiquing is easier than creating.
Example of
greater efforts to support (non-cumulative): much greater support at lower
levels; higher level have been problematic.
Practice – Bloom and IO’s
Students can
identify correct
punctuation marks in a writing assignment. (Knowledge/Application/Analysis) [Highlighting = More correct]
Students can distinguish facts from opinions in
newspaper editorials. (Analysis)
Students can add previously unseen proper
fractions. (??)
Given a
two-page essay, the student can identify
assumptions basic to the author’s position. (??)
The student
will correctly spell the word ‘mountain.’ (Knowledge)
Who wrote
‘The Great Gatsby?’ (Knowledge)
If a farmer
owns 40 acres of land and buys 16 more, how many acres does he have?
(Application)
Which musical film performer was probably the best athlete? (Evaluation)
Align:
planning instruction, delivering instruction, assessing outcomes.
How do we align all 3 phases?
Write good
instructional objectives (IO’s) in the planning phase.
Problem:
people confuse IO’s with other ideas/concepts: expectations, educational
goal/global objective, general objective, and instructional objective/specific
objective.
Expectations
Teacher
beliefs about what students are capable of doing.
Based on
observations/collection of data about student behavior. (Family members who’ve
been former students lead to generalization about other family member who is
current student). E.g. previous achievement, aptitude, motivation, interest.
Initial
expectations – be wary!
Formed
before the teacher meets students.
Formed
during initial interactions.
Based
on student attributions.
Result
in differential treatment.
The
Progression: broad to narrow
Instructional Objectives (Specific
Objectives)
Specific
statement of what students will be able to do as a result of instruction.
Must be directly measurable!
Relates to
specific instructional tasks: directly tied to classroom instruction.
IO’s are
useful for designing and planning daily lessons activities.
Observable
behavior (action verb): directly measurable, some evidence of ability.
Conditions
for display of behavior: necessary or allowable materials, length of time.
Performance
level for mastery: the percentage correct or tasks to perform in order to
master or pass.
Parsimony in
objective and assessment: keep it simple for yourself and students. There are
multiple methods for assessing many objectives, un-necessaryily difficult ones
may complicate the objective with the task.
Match the
objective to the assessment:
Instruction Objectives are: a statement of what students should
achieve at the end of a unit of instruction. Provide organization and
clarity of instruction. Help better align the three (3) phases of instruction.
Help you talk about what’s going on in your classroom in terms of what
students are learning. Are useful for communication with others.
Instructional Objectives should not: Activities - IO’s define specific
skills for each activity you’ve planned. May apply to one or more specific
activities. Standards – though good IO’s are often based on standards. Stated
in terms of what the teacher will do. NOT “The teacher will teach
students the rules of basketball.” In terms of student behavior.
