Tuesday, May 15, 2012

U. of AZ Graduate-Level Education Policy 558 (EDP 558) Notes

Textbook for the course:
Educational Testing & Measurement
ISBN: 978-0-470-52281-3


NOTES:


Thursday, May 16, 2012

The Progression is an inverted pyramid: Educational goal on top, general objective next, then IO last.

Educational Goal (Global Objective)
Broad statements of what students will be able to do. Framed within a relatively broad content domain. Tend to be lofty or noble. Take a very long time to actually accomplish. May not be in one day, could be in a lifetime. These are usually not directly measurable. Provide a good starting point, but not specific enough to be an IO.

General Objective
More specific than educational goals. Framed in terms of student behavior. Tied to actual classroom instruction. Usually called “state standards.” National standards? Kept switching back and forth. Still too broad to make good IOs. Useful for curriculum design and planning large instructional units. Generally, where students by the end of the unit.

Structure of State Standards:

1)    Performance Standards (proficiency levels)
a.     Measure of how well student is able to demonstrate the learning.
b.     Describes the level of performance that students should achieve.
c.     NCLB calls these academic achievement standards.
d.     AZ Dept. of Ed. ex: falls far below, approaches, meets, exceeds (FAME).
2)    Content Standards
a.     General statement that describes what students should know/skills they should be able to do.
b.     NCLB call these academic content standards.
c.     Usually pretty general.
d.     AZDE ex: Math, Grade 2, Strand 1, Concept 1: Understand and apply numbers, ways of representing numbers, and the relationships among numbers and different numbers systems.
e.     English, Language Arts and Math.
f.      Organized by:
                                               i.     Domain
1.     Groups of related standards and clusters
2.     Fifth (5th) grade math example: # and operations in Base Ten (10) – 5.NBT
3.     Cluster (NCLB calls these content standards)
a.     Groups of related standards
b.     Ex: Understand the place value system.
4.     Standard (NCLB = benchmarks)
a.     Ex: 4. Use place value understanding to round decimals to any place.
b.     Abbreviated for this class 5.NBT.4
                                                                                                     i.     Consecutively numbered no matter how many clusters.
3)    Benchmarks – “the demonstration”
a.     Specific skills within the broad content standards.
b.     Students must do these to demonstrate they met the standard.
c.     Specifies what the student is to actually perform.
d.     Similar to IOs, but not designed specifically for classroom use.
e.     AZ calls these performance objectives (PO).
                                               i.     AZDE ex: Math, Grade 2, Strand 1, Concept 1, PO 1. Express whole numbers 0 to 1,000 in groups of hundreds, tens and ones using and connecting multiple representations.
Points to Clarify
1)    Every state has to have these three (3) levels, but the terms vary by state.
a.     E.g. Some states use the term “proficiency levels” for “performance standards” and use “performance standards” for benchmarks.
b.     Content standards do not mandate what happens in the classroom.
                                               i.     They are what is to be learned NOT a way of learning!
                                             ii.     They provide a broad framework for curriculum.
c.     Districts often provide their own specific benchmarks as well.
                                               i.     Specific to district needs.
                                             ii.     Assessed using a benchmark test.
ABCD’s of objectives
A)   Audience: who is doing the behavior?
a.     Instructional objectives are about the students.
B)   Behavior: what is to be done?
a.     How the behavior is demonstrated.
C)   Conditions: what are the constraints or allowances?
a.     Permitted or not materials (calculators, dictionaries), time to complete, alone or in groups. 
D)   Degree: how well?
a.     Determination of success or failure, moderate levels of success.
E)    Everyone: SHOULD AT LEAST CONSIDERED WITH ALL
            ‘OBJECTIVES.’
To Start
1)    Look at the standards in your area.
2)    Create a content outline for the unit.
a.     Components of the unit in outline form
3)    Think about what a substitute about what a substitute teacher would need
a.     Even if you plan to be there throughout the unit
b.     It’s a good practice for keeping it clear – remember that others may ask to see these! i.e. Parents, administrators.
c.     Focus on the students and their performance – NEVER on the teacher.
d.     Focus on the learning product …
Writing
1)    Beginners should use a general stem (“SWBAT” – Students will be able to …)
                                               i.     Reminds you of who the audience of these objeves are.
                                             ii.     Feel free to omit general stems once you aree comfortable with the format.
b.     Focus on observable and measureable behavior (use good action verbs).
c.     Include examples if using general or vague terms.
d.     Write a few at different Bloom’s levels and determine which are best.
                                               i.     Remember that higher-level skills include lower-level skills.
Don’t Do:
1)    Use double-barreled objectives (i.e. an objective that includes two
            goals/objectives in one ‘objective’ in benchmark objectives.
2)    Get too specific or general
3)    Include objectives that seem like they will be difficult to teach or
            measure.
a.     These are education goals.
b.     Fine to have, just don’t call them IOs.
What do you think?
1)    Students will learn how to read technical papers. (Should be: Students will be able to identify common parts of a technical paper).
2)    Students will be able to read the following paragraph out loud with fewer than four (4) errors. (TOO SPECIFIC, IT’S A SPECIFIC TASK)
3)    Students will be able to identify at least four (4) of the major battles of Civil War and identify the causes of the Civil War. (DOUBLE-BARELLED)
Formal vs. Informal Assessment
1)    Formal: usually planned in advance.
a.     Systematic (e.g. design issues/grading criteria)
b.     Relatively infrequent.
2)    Informal: less structured
a.     Relatively frequent.
b.     Typically not graded.
c.     Common strategy – asking questions
                                               i.     Consideration for question asking
1.     Stay on content
2.     Avoid generic or vague questions
3.     Take environment into account (Year in school/Grade level)
4.     Be aware of your patterns (bias) in calling on students.
a.     Both male and female instructors call on boys more often than girls.
b.     Action Zone: Group that interacts with the teacher, one that does.
5.     Be aware of wait time
a.     Low level Questions: 3-5 seconds
                                                                                                     i.     Adopting a required wait time may produce more student participation.
b.     High level Questions: 10-30 seconds
6.     Reword questions if students seem confused.
7.     Use follow-up questions.
a.     To allow others to answer or to get students to delve deeper on a topic.
8.     Be careful how you respond to incorrect answers.
a.     Mediate. “That’s not how we use it in this context.”








Tuesday, May 15, 2012

Analyzing standardized test…
1) Cut-offs change from year-to-year
2) Broken down by: Strands & concepts
3) Norm-reference test (NRT) [i.e. Stanford 10]
            • Scores are 1-9. Ranges are 1-3, 4-6, 7-9
Arizona Interment for Measurement Standards (AIMS) was not required for 2011-2012 cohorts to graduate high school. Could possibly be required for 2012-13 cohort to graduate from high school.
High-Stakes Testing (HST) proponents: ensures all students are held to the same standards and that everyone takes learning more seriously.
Gives an = way to measure for making comparisons and determining accountability.
HST is another tool in the assessment toolbox to assess.
Some research says it allows learning to transfer to other measures and/or areas.
2004 polls (No Child Left Behind – approved in 2002, enacted 2004): > 70% support some HST practices.
HST used as part of educational reform and accountability.
Most cite need for objective measures.
HST critics:
Emphasizes rote learning!
Too much time is spent on preparing students of the test rather than on meaningful learning.
May not be the most appropriate way to measure learning. Don’t measure creativity, freedom of expression.
May penalize teachers who work with academically struggling students.
Some indications suggest that HST has cultural bias.
HST is often a misused tool.
Increased cheating can happen with HST. Scandal in Georgia (1-2 years ago, teachers, administrators were fired.)
Some research correlates (no correlation) HST with increased dropout rates.
HST scores correlated with decreased property (poverty in school districts) values.

American Educational Resource/Research Association’s (AERA) criteria for HST
No single test should be used for decision-making.
Provide adequate resources and opportunity to learn the material before test.
Validation’ for intended use to measure ‘intelligence’ and there are sufficient indicators that are reliability.
HST ‘should’ provide full disclosure of potential negative consequences.
The test and the curriculum ‘should’ be aligned.
Validation’ of passing scores and achievements for schools and pupils are an important result of HST.
Meaningful remediation courses/course work for failing examines (not as many or as intensive as in the past, because the budget for these programs was cut).
Attention on HST is given to those with language differences.
Attention to individuals with learning disabilities. (Varies from district-to-district.)
Five years ago a school allowed learning-disabled students to use calculators on the math portion because of their Individual Educational Plan (IEP).
Adherence to testing rules for whom will be tested.
Teaching to the Test?
Often gets misinterpreted as “teaching the test.”
Instruction is driven by the test’s content domain.
So, the test is not driving your instruction, your objectives are:
            Teaching general test skills.
            Focused on learning processes/life-skills.
Teaching the test
Instruction is driven by what items will be on the test, and in some cases the direct content that will be on the exam.
A test shapes how teachers instruct pupils.
Teaching revolves around the test.
Teachers who have to ensure their students’ success on HST should ‘Teach to the Test,’ but most ‘Teach the Test.’
Improper Prep Techniques
Don’t give them exact test they’ll be taking.
Don’t give them a parallel form of the test they’ll be taking.
Don’t find out the answers to the items on the test and give them to your students.
Don’t deliberately deceive your students about the directions, intent, or content of the tests.
Proper Prep. Tech.
Evaluate your own biases and be objective about the need for preparing.
Be knowledgeable about the test, purposes, and scoring.
Familiarize students with the format of the HST.
Practice with released sample items.
Prepare them for the length of time they will be testing. Tell students to bring books/activities to do if they finish the test early.
            Don’t test for four (4) days straight, but give them practice testing for 60 or 90
minutes at a time.
            Many teachers allow, food, free time after testing to break it up.
Teach general test skills.
Teaching Test Skills
Definition: Skills that allow students to more efficient and effective test-takers.
Suggest that students:
Jot down any mnemonics/reminders before taking the test, if there is free space.
Skim entire test.
Answer any you know immediately.
There may be cues/reminders in later items to assist with answering earlier items.
Make note of item weights. (1 pt. item vs. 10 pt. item, focus on 10 pt.)
Make a mark on items that you need to think about and come back to.
With multiple choice, eliminate response choices that you know are incorrect.
If there is a penalty for guessing, you should try anyway if you can eliminate even
a single distractor. Can lose extra points if the test ‘graders’ if they determine if a
student is ‘guessing.’ Gender distinction: Men are more likely to guess on a test
question and they often get higher scores for ‘guessing.’
With true/false, underline or make note of parts that make the item false.
Use poor test creation to your advantage – look for grammatical or other parallels
between stems and response choices.
Review the test before turning it in.
Do NOT change your first answer unless you completely misread it the first time. Subconscious forces typically direct you to the correct response.
Make sure you’ve answered everything.
Sit next to smart kid (JOKE!)
Teacher said to use the same pencil they take notes with because it has all the answers.
Bring candy to test, possibly.
Sit in the same spot you do in class when taking a test.
Keep a similar routine! Go to the bar, if you go to the bar before a test. (LOL)
           
Know reliability vs. reliability for test

Creating & Measuring Objectives (Ch. 6)
-       Explain the purose of Bloom’s Taxonomy and why it is impt.
-       Planning assessment
-       Instruction Objectives (I.O.)

Create well-written I.O.’s for use in classrooms.

Bloom’s Taxonomies (1954)
Bloom is one of the most significant influences on modern education. Author of “one of the most widely cited yet least read books in American education.”
What’s taxonomy: a way of classifying something hierarchically (low to high)
Bloom organized behavior
-       psychomotor
-       affective
-       cognitive – this is where we focus most
Pyschomotor
Present at birth, uncontrolled, unlearned responses.
Basic-fundamental movements: More skilled, intentional movements.
Perceptual abilities: sensory information, sorting through stimuli.
Physical abilities: involve flexibility and strength.
Skilled movements: complex movements toward a complex goal.
Non-discursive communication: non-verbal communications.
Use of psychomotor taxonomy
Not as useful for most of our purposes.
More informative for those working with children who:
            Are younger
            Are in physical education
            Have physical difficulties
May create objectives related to classroom behavior that stem from psychomotor.
Teachers should watch for difficulties in any of these areas.
Bloom’s affective taxonomy
Hierarchy of methods to measure feelings or motivations.
Attention (receptiveness): aware, willing, and selective perception of the transmission of information.
Responsiveness: active, willing, satisfying (to the student), and participation.
Valuation: acceptance, preference, and commitment expressed.
Organization: recognition or the value system and prioritization among values.
Behavior according to values: avoidance of hypocrisy, general or within identified limits.
Use of Affective taxonomy
The hope is that students value learning and internalize these values.
            Can help students see the inherent benefit to the material. Help them in life, not
just with academia.
Encourage alignment of behavior and beliefs.
Move beyond external motivators (grades).
Be able to identify and defend their belief systems.
A lot of crossover of Bloom’s taxonomy, it can be cognitive; but more strongly is identified in an affective manner.
Bloom’s Cognitive Domain
Use Bloom’s taxonomy to organize instructional objectives.
Ideal teaching should focus on higher-level cognitive skills.
            But learning at higher levels depends on prerequisites knowledge.
            Prerequisites knowledge is the result of having attained lower-level objectives.
Lower-level cognitive skills
            Knowledge
                        Using recall and recognition
                        Accessing rote memory of facts, terms, basic concepts, sequences,
classifications, principles, theories.
            Comprehension
                        Grasp meaning/understanding
                        Demonstrated by organizing, comparing, interpreting, describing, and
summarizing main concepts.
Higher-level cognitive skills
            Application
                        Using knowledge in novel situations.
                        Requires applying knowledge in different ways.
            Analysis
                        Breaking down a complex idea into parts.
                        Understanding relationships and causes.
                        Making inferences and identifying support for those inferences
            Synthesis (2nd to the most complex)
                        Putting together several ideas to form new patterns.
                        Finding new patterns or alternative solutions.
                        Often demonstrated by production of a new product.
            Evaluation
                        Evaluating and assessing complex ideas
                        Presenting and defending judgments and positions
Mnemonic Devices
            How can you remember the order: Knowledge, Comprehension, Application,
Analysis, Synthesis, Evaluation?
                        One way: Kids Can Always Achieve Standards-based Education
Revised Bloom’s Taxonomy (2001)
Bloom (1954) was interested in classifying levels of behavior.
New taxonomy was written for a different audience and purpose (K-12 educators).
            Increased focus on categorizing objectives and then aligning activities and
assessments with those objectives.
            Increased use of subcategories.
            Tendency to reframe in terms of verbs – learning is action-oriented.
Two New Dimensions
Knowledge becomes it own dimension à remembering goes into cognitive domain.
Cognitive domain revised to contain other six (6) levels: evaluating comes before Creating (Synthesis) now. Critiquing is easier than creating.
Example of greater efforts to support (non-cumulative): much greater support at lower levels; higher level have been problematic.
Practice – Bloom and IO’s
Students can identify correct punctuation marks in a writing assignment. (Knowledge/Application/Analysis) [Highlighting =  More correct]

Students can distinguish facts from opinions in newspaper editorials. (Analysis)

Students can add previously unseen proper fractions. (??)

Given a two-page essay, the student can identify assumptions basic to the author’s position. (??)

The student will correctly spell the word ‘mountain.’ (Knowledge)

Who wrote ‘The Great Gatsby?’ (Knowledge)
If a farmer owns 40 acres of land and buys 16 more, how many acres does he have? (Application)
Which musical film performer was probably the best athlete? (Evaluation)
Align: planning instruction, delivering instruction, assessing outcomes.
How do we align all 3 phases?
Write good instructional objectives (IO’s) in the planning phase.
Problem: people confuse IO’s with other ideas/concepts: expectations, educational goal/global objective, general objective, and instructional objective/specific objective.
Expectations
Teacher beliefs about what students are capable of doing.
Based on observations/collection of data about student behavior. (Family members who’ve been former students lead to generalization about other family member who is current student). E.g. previous achievement, aptitude, motivation, interest.
Initial expectations – be wary!
            Formed before the teacher meets students.
            Formed during initial interactions.
            Based on student attributions.
            Result in differential treatment.
The Progression: broad to narrow
Instructional Objectives (Specific Objectives)
Specific statement of what students will be able to do as a result of instruction.
Must be directly measurable!
Relates to specific instructional tasks: directly tied to classroom instruction.
IO’s are useful for designing and planning daily lessons activities.
Observable behavior (action verb): directly measurable, some evidence of ability.
Conditions for display of behavior: necessary or allowable materials, length of time.
Performance level for mastery: the percentage correct or tasks to perform in order to master or pass.
Parsimony in objective and assessment: keep it simple for yourself and students. There are multiple methods for assessing many objectives, un-necessaryily difficult ones may complicate the objective with the task.
Match the objective to the assessment:
Instruction Objectives are: a statement of what students should achieve at the end of a unit of instruction. Provide organization and clarity of instruction. Help better align the three (3) phases of instruction. Help you talk about what’s going on in your classroom in terms of what students are learning. Are useful for communication with others.
Instructional Objectives should not: Activities - IO’s define specific skills for each activity you’ve planned. May apply to one or more specific activities. Standards – though good IO’s are often based on standards. Stated in terms of what the teacher will do. NOT “The teacher will teach students the rules of basketball.” In terms of student behavior.