Item Response Theory « Result #1 on Nov 18, 2006, 10:56am »
The computer that grades the FCAT test will not give full credit for a CORRECT answer if it "thinks" the answer was a guess!!! Now, tell me, how can a computer know if your guessing!!?? The program the computer uses to do this is based on something called Item Response Theory.
Again, how the heck can a computer attempt to figure out all the stuff that is going on in a student's mind when that student is reading, thinking, and answering questions!! That seems a little science-fiction doesn't it?
For example, last spring after my students took the FCAT I asked them to tell me (if they could remember) some things they were thinking about during the test that had nothing to do with the test. Here is a list of some of those things:
"somebody farted" food/being hungry homework for other classes "how does impotence work?" "liking the guy behind me, and I wore the dumbest shirt" being beat up after school "my pencil lead broke and I began wondering what the inside of a school pencil sharpener looks like" a poster on the wall showing Macbeth fighting "the story kept reminding me of my dad and I kept thinking about last fall" "my pits were sweating...bad" "my answers were falling into a pattern, and I didn't want to break that pattern even if I knew the answer was wrong" (!!!!)
All these responses came from ONE student!!! A student who does well in my class. What's more she told me she thought all of this in the span of a couple of minutes!!!
How can a computer program attempt to understand these sorts of mental processes and then attempt to quantify how these wonderings produce what it deems "knowledge" or "guessing"????
Reading Level of the FCAT « Result #2 on Nov 18, 2006, 10:38am »
The FCAT is supposed to test 10th graders, correct? Then why are half the readings passages written at 11th grade, 12th grades, and sometimes even college reading levels!? In 2005 the Florida Department of Education released the 2004 FCAT test. This released test contained six readings in the Reading portion: one poem and five prose pieces.
On a whim, I decided to do a reading level analysis on the Reading portion. I went to the DOE’s website (www.fldoe.org) and located the released test. I transferred the PDF files into a Word document (sometimes typing whole passages!) and performed a Flesch-Kincaid reading level analysis on the readings using the options available on Microsoft Word. I took an analysis for each reading as whole, as well each paragraph within each reading. The results for the five prose readings are below.
"Bike-Friendly Communities" - · The entire piece is an 11.4 grade/reading level. · All but one paragraph is written at a 12th grade/college reading level. · The "bullets" included in the reading are a 9.7 reading level · IN GENERAL: The combination of the bullets and the single paragraph not at the 12th grade/college level reading assist in reducing the overall reading level for what is essentially a 12th grade piece. Significant portions of the reading are at the reading level required to read the Harvard Law Review (generally in the 30's on the Flesch Reading Ease Test).
"What's Your Best Time of Day?" · The entire piece is a 9.7 grade/reading level. · However: nearly 50% the reading is above the intended reading level the test is designed to assess. Two paragraphs are written at a 12th grade/college reading level. . Significant portions of the reading approach the reading level required to read the Harvard Law Review. · IN GENERAL: The inclusion of a third grade reading level concluding paragraph and two sixth grade reading level transition paragraphs (the longest of which is two sentences) skews the total results, disguising the true reading level for the main information of the piece.
"The Origins of Baseball" · The entire piece is an 8.6 reading level · Two paragraphs are written at an 11th grade reading level. 44.5% of the questions dealing with this reading ask for general information from these two paragraphs, and almost a quarter of the questions ask for specifics from these same two sections. · IN GENERAL I find it odd that such a large percentage of the questions for this piece come from the two paragraphs that are written above the intended grade level the test is designed to test.
"After You've Stood on the Log at the Center of the Universe, What Is There Left to Do?" · The entire piece is written at a 3.6 reading level.
"Women Who Shaped the Constitution" · The entire piece is a 11.7 reading level · Nearly 50% of the piece is at a 12th grade/college reading level . Significant portions of the reading are written at the same level as the Harvard Law Review
Again, if this test is intended to test 10th graders, why is much of it written at a reading level that would challenge a 12th grader!!??
I would ask that you do the same as I did, perform a reading level analysis of the test, and discover for yourselves these odd findings.
Commissioner's Letter « Result #4 on Mar 27, 2006, 8:57pm »
Dear Colleagues:
As spring approaches, I hope this e-mail finds you well and preparing for a well-deserved break.
Many of you have expressed your views and concerns over the recently adopted State Board of Education rule on performance pay. Let me share some additional information with you.
This week, I will appoint an "E-Comp" advisory committee made up of teachers and administrators. This group will serve as a forum to discuss implementation issues and options available to school districts as they comply with the law. Members will advise me on ways to ensure that districts have as much flexibility as possible, as well as share innovative ideas to improve the overall annual evaluation systems across the state.
To summarize, the four-year-old law requires the following basic elements:
* Districts must provide a 5% annual bonus to its most outstanding teachers. (The rule stipulates that a minimum of 10% of each district's teachers must be identified as outstanding.)
* Student learning gains must be a “primary” component in the evaluation of outstanding teachers. ( The rule identifies the FCAT as the most reliable and valid measure of student learning gains in reading, mathematics and eventually science .)
* All teacher evaluations are to have seven components, including student learning as a primary one. (The rule defers to district annual evaluations for non-FCAT teachers, counselors, reading coaches, media specialists, etc.)
* Every teacher's pay must be based, in part, on performance. (The rule requires that the annual evaluation be capable of differentiating levels of performance to be incorporated into annual pay increases.)
* Not addressed in law, the rule prohibits burdensome requirements that teachers submit to an arduous application process in order to be eligible for a performance-based bonus. Previously, these measures only ensured that few were ever rewarded.
As a result, many districts will have to redesign their annual evaluation instruments and/or processes. As a profession, we should have professional and meaningful evaluations applied in a fair and objective manner. Principals will be offered training to ensure annual evaluations are fair and provide meaningful feedback on teacher performance.
Any bonuses awarded this year will be based on your district's existing performance-pay plan. However, bonuses for the 2006-2007 school year will be based on negotiated and state approved plans that reflect the minimum framework outlined in the new rule.
I will continue to send you regular updates on this important issue.
The Emails (Part 4) « Result #5 on Mar 27, 2006, 8:42pm »
(Thursday, January 26, 2006)
Dr. Orr,
Thank you for your response regarding the FCAT and its research. However, my question has still not been answered.
I have read the material you have provided me before. It may be stated that the FCAT is valid and reliable within its own epistemological/pedagogical paradigm. This is stated in the summary of the validity and reliability section of the Assessment and Accountability Briefing Book:
"The evidence of reliability and validity supports the claim that FCAT is technically sound and meets or exceeds the professional standards for standardized achievement tests."
The FCAT may satisfy its own requirements within the realm of standardized achievement tests. However, and this is in spite of the DOE's "cover-your-butt" statement regarding how teachers shouldn't "teach-to-the-test," from my interviews and talks with professors and teachers from around the state, it seems that the FCAT is no longer merely a test, rather it is becoming a curriculum. It has reduced the constantly contested, ever-changing and immensely huge concept of "intelligence" to a handful of "essentials." And only these “essentials” are tested, and, often, only these “essentials” are taught. If one cannot master these essentials, one lacks the intelligence to graduate from high school. So states the FLDOE. Thus, the FCAT has become an exclusive “system of knowledge,” which cannot allow other systems to operate within it or about it.
Let us say I am a teacher, and I'm not supposed to teach to the test. So, I only give 5% of all instruction time to the FCAT and the curriculum that has grown up around it, devoting the remaining instruction time to other "systems of knowledge," perhaps one based on developing my students' emotive intelligences, or one based on some post-modern thought of Barthes in which the intentionality of the author does not exist, thus there goes "main idea" of a work, along with "author's purpose," etc. With Barthes, these two things may not exist. And, certainly, this is a fair approach to take, for most of our culture today is based on post-modern thinking, with the Internet itself being the absolute manifestation of this school of thought.
To further show the limits of the FCAT, lets us say that instead of assessing my students on their Barthes-informed reading with a paper-n-pencil, multiple choice test, I allow my students to develop their own personal multiple-intelligences by interpreting the text using the medium of Canvas, Music, Dance, Poetics, Database construction, etc. So, that my students have never taken a paper-n-pencil, multiple choice test (which isn't all that crazy seeing as how in the "real world" one NEVER takes a paper-n-pencil, multiple choice test to show they can get a certain task done. Rather, they simply get the task done).
Now, as this teacher, I have done my required job by setting aside some instruction time to teach the contents of the Sunshine State Standards, but I have also included other systems of knowledge and assessment. How will my students do on the SSS-backed FCAT compared to a teacher who devotes 90% of her instruction time to the FCAT curriculum? Which group of students constructed greater amounts of personal, meaningful knowledge in their lives?? In this scenario, is the FCAT a reliable and valid assessment tool?? I think the answer to all of these questions are obvious.
Thus, back to my original question/quest.
Whether or not the FCAT, as a standardized test, is valid and reliable, what research does the FLDOE have to demonstrate that the FCAT (or any other standardized test) is the fairEST, MOST reliable, and valid demonstrator of personal, meaningful knowledge constructed in a student’s mind?
To put it another way, of all the different approaches to assessment and curriculum construction, what research does FLDOE use to prove that a standardized test approach is the fairESt, MOST valid and reliable way to go?
Again, the research I am seeing, the research mentioned to Richard Hardy in previous emails, seems to indicate that of all the approaches the standardized test one is nowhere near the top of the list.
Again, thank you for your patience,
Concerned Parents ************************************************************************** February 7, 2006 Dear Mr. McCloud: Thank you for writing again to express your views about the Florida Comprehensive Assessment Test® (FCAT), content standards, and the general nature of statewide assessment programs. This office has attempted unsuccessfully to provide you with the information you have requested on two occasions, and you express in your letter that Richard Hardy in the Office of School Improvement has also been unable to satisfy research requests you have made. Although we have attempted to provide adequate representation of the FCAT program, it is now clear from your recent letter that we have been unable to satisfy your requests because you and the State hold philosophical differences in relation to statewide assessments. Thank you for taking the time to write. We are sorry we are unable to prove that the FCAT is fair, valid, and reliable, but we hope you continue to participate in statewide assessment discourse with educational colleagues around the country. Sincerely, Cornelia S. Orr, Ph.D.
******************************************************************************* (Wednesday, February 8, 2006 ) Dr. Orr
You state: “We are sorry we are unable to prove that the FCAT is fair, valid, and reliable…” Again, this is not my main issue. My main issue is, is the FCAT the MOST fair, valid, and reliable tool of measurement?
I hold no "philosophical difference" to the DOE's approach (i.e. At this point in time, I hold no contention to the fact that within the paradigm it has created for itself the FCAT is valid, reliable and fair). I am simply following the research made available to me by the avenues I have mentioned to you in previous emails, that is, research which suggests that the standardized test approach is not the MOST valid and reliable means of curriculum construction and assessment/evaluation. I was hoping that Florida would be able to provide me with comparable research across a wide spectrum of assessment and evaluation approaches, from which the DOE then selected the standardized test approach because it was the MOST (please allow me to reiterate that term once more: MOST) valid and reliable. It appears the FLDOE has not done this research, or at least is unable to provide a parent with it at this point. Which seems a bit odd, because if I am able to get this research, the FLDOE should be able to so as well. But, you are correct, it seems to be the fact of the case that for whatever reason the FLDOE does not have this most basic of research.
I pose a rhetorical question: If one wished to get to the corner store for the daily paper how would one do it? Walk? Well, that is certainly a valid and reliable way. Bike? That too seems valid and reliable. Drive? Take the bus? Skateboard? Scooter it? Etc. There are many approaches. Let us say in our example, and I apologize if it is a bit simple, but: let us say that the standardized test approach is walking. Or it could be the skateboard. Or the car. It doesn't matter, because if we wanted to find out the M O S T valid and reliable way we would have to compare ALL the approaches, we would have to research all the approaches to determine the most valid and reliable. Again, I apologize for the basic comparison, but I think (hope) I'm getting my point across. I hope you can see how unsettling it is for a parent to find out that his child is in a system in which no one knows what the best approach is because no one wants to compare options! Is this not bizarre? That seems like the first step in coming up with any solution!! Commissioner Winn promises Florida's students the "best education possible," yet you have no comparable research that suggests this is the case? How can he make what may actually be (with the research I have) an empty promise to millions of parents?
Concerned and Confused ********************************************************************************
(February 14, 2006) Dear Mr. McLeod: I am sorry you are dissatisfied with our last reply to your third request for FCAT research. I indicated that you hold philosophical differences with the Department of Education because you have been dissatisfied with the research my staff and Richard Hardy have collected and provided to you. If you are seeking additional research that debates the nature and impact of assessment, I encourage you to read the periodicals mentioned in previous correspondence. They hold the type of research and pedagogical discourse you are seeking and although the Department of Education does not conduct the research these periodicals debate, we have used these periodicals as guides as we continue to make improvements to our program. Since you are dissatisfied with the type of research this office has provided, I believe these periodicals will contain the type of research you are seeking. Thank you for your interest in the statewide assessment program. Sincerely, Cornelia S. Orr, Ph.D. Director Assessment and School Performance ******************************************************************** (February 14, 2006) Dr. Orr,
Again, thank you for your response. Some of these journals and resources I was familiar with and some were new to me upon your reference to them some weeks ago. In light all of the FLDOE's responses, is it safe for me to say then, that the answer to my original question -can Florida provide me with the authors and names of research studies demonstrating that the standardized test approach to assessment and evaluation is the MOST fair, reliable, and valid?- is NO, the FLDOE cannot provide me with the names of such research studies. ********************************************************************* No Response was received. The following email was sent after I read the Commissioner Winn letter to Florida teachers. ******************************************************************** (March 26, 2006)
Dr. Cornelia S. Orr,
Hello, we exchanged emails recently regarding the lack of research the Florida Department of Education possesses to show that the standardized test approach, out of ALL options and approaches, is the MOST fair, reliable and valid form of assessment.
During these email exchanges, the FLDOE attempted to steer the discussion to the merits of only one approach, namely the one it has run with, the standardized test approach, essentially ignoring my original question, which was not answered until almost four months later in a resounding, No.
Recently, I was speaking with a teacher about the latest round of FCAT testing (Feb-March 2006). At the end of our interview I was shown an email sent out by Commissioner Winn. In the email, Commissioner Winn was explaining the workings of the new plan to attach pay advancement to FCAT scores. In the email Mr. Winn stated:
"Student learning gains must be a “primary” component in the evaluation of outstanding teachers. (The rule identifies the FCAT as the most reliable and valid measure of student learning gains in reading, mathematics and eventually science.)"
Obviously, Dr. Orr, I was shocked to read this statement, for it is, after all, exactly what I have been attempting to get from the FLDOE!! Commissioner Winn categorically states that the FCAT is the M O S T reliable and valid measure of learning!!!!! What research does he have to back that statement????
But, the FLDOE has already answered that question; the FLDOE has no research that has compared various assessment approaches. Yes, within the arena of a single approach, of ONE method, the FCAT, the DOE has indeed determined that it can safely state the development of this exam is reliable and valid, but the DOE cannot claim that the FCAT is the MOST reliable and valid measure because it doesn't have the research. Is this statement correct? If not, please, AGAIN, allow me to have the names of the research studies which would make this statement false. The research that apparently Commissioner Winn has, but the DOE is refusing to give to us.
The Emails (Part3) « Result #6 on Mar 27, 2006, 8:41pm »
(Thursday, December 8, 2005 )
(Again) Mr. Hardy, Will(can)the state of Florida provide me with a list of research studies/projects it uses to back it's stance on the development of the SSS and the FCAT as the fairest, most accurate measurement tool of student achievement? I see there are two situations here: 1. The state of Florida has these research studies and for some reason you are not giving it out to parents who are asking for it, parents who are extremely concerned with our son' success both in school and out of school, OR 2. The state of Florida does not have any research that it has based its decisions on, rather it simply closed its eyes and got in line behind the "standards" rhetoric, which a strong majority of research and professors of Education claim is dangerous to the development of children. Which of the two is it? Is there a third option that I am missing? Can I please, please, please, please just have a list of research studies Florida can cite as its reason? Concerned with the lack of information available, G.
******************************************************************** (Thursday, December 8, 2005)
Mr. McCloud: I have provided you with the information and research that I can personally substantiate regarding your request. Our office does not create statutory educational policy and the development of both the Sunshine State Standards and FCAT occurred long before my tenure with the Department of Education had begun. Both the Sunshine State Standards and FCAT are currently state statutes. Our office is prepared to assist in educational requests relating to public schools with topics, issues or concerns related to the statutory language which currently exists. If your focus is to have statutory language amended or abolished regarding the Sunshine State Standards and/or the FCAT, your appeal would be more effective if directed at the State Legislature or Governor’s office. Thank you, Richard Hardy ************************************************************************** At this point I contacted Commissioner Winn's Office:
(Wednesday, January 04, 2006) To: Commissioner Winn Hello, A couple of weeks ago I requested documentation of the research studies the state of Florida uses to support its creation and implementation of the FCAT as the most reliable, valid and fair tool for measuring actual student achievement in our state. I have yet to receive a reply. I am hoping this information can be provided to me. Thank you. Concerned Parent
************************************************************************** January 4, 2005
Dear Mr. McCloud:
Thank you for contacting Education Commissioner John Winn for information about the Sunshine State Standards (SSS) and the Florida Comprehensive Assessment Test® (FCAT). Commissioner Winn received your message, and I have been asked to respond to your concerns about the FCAT on his behalf. I would first like to provide you with an explanation of the history of the SSS and FCAT and then, with helpful resources to assist you and your wife as you research the FCAT and the content benchmarks it assesses.
The Florida Commission on Education Reform and Accountability began conceptualizing the FCAT well before the first test was administered in 1998. In 1995, the Commission recommended procedures for assessing student learning in Florida that would raise educational expectations for students and help them compete for jobs in the global marketplace because seven percent, that is 158 public schools, were identified as performing at a critically low level. The State Board of Education adopted the recommendations, called the Comprehensive Assessment Design, in June 1995. The Design specified the development of new statewide assessments to address four broad areas described in the first four standards of Goal 3 of Blueprint 2000, a reform plan for Florida’s public schools. These four areas have been generally referred to as reading, writing, mathematics, and creative and critical thinking. In addition, the Design required that educational content standards be developed and adopted. Subsequently, the Florida curriculum frameworks, also called the Sunshine State Standards, were developed and adopted by the State Board of Education. The frameworks and standards established guidelines for a statewide system that incorporated assessment, accountability, and inservice training components.
The FCAT was designed to meet both the requirements of the Comprehensive Assessment Design and the rigorous content defined by the Sunshine State Standards. The FCAT measures the content specified within the strands, standards, and benchmarks of the Sunshine State Standards and does so in the context of real-world applications. Initially, the FCAT was designed to assess reading, writing, and mathematics at four grade levels so that each subject was assessed at all levels of schooling: elementary, middle, and high. With legislative approval of Governor Bush’s A+ Plan in 1999, the FCAT was expanded to include grades 3 through 10, and in 2003, the Grade 10 FCAT replaced the High School Competency Test (HSCT) as the graduation examination for earning a high school diploma (Florida has had a graduation examination since 1976).
The development of FCAT Reading and Mathematics questions began in 1996 when the State Board of Education adopted the Sunshine State Standards and the Legislature authorized and funded the statewide assessment program as a means of determining the extent to which our children are learning the Sunshine State Standards. On November 24, 1998, the State Board of Education, chaired by former Governor Lawton Chiles, approved the designation of five achievement levels (levels 1 through 5) for FCAT score results. The proposed levels were reviewed by:
· A group of school districts, college administrators and teachers’ association representatives. · Two citizen business groups in Orlando and Tallahassee – School district superintendents, assistant superintendents, and testing coordinators · Educators/citizens at three public hearings · Department of Education staff and the Commissioner of Education.
In this context, Governor Bush fought for legislative approval of the A+ Plan for Education which built upon the existing five achievement levels by changing them from levels 1 through 5 to grades A through F to give the public a greater understanding of what the levels mean. Results for students, schools, and districts on the additional tests were reported for the first time in 2001.
In your letter, you request a list of research studies and their authors that the Florida Department of Education uses to back the creation of the Sunshine State Standards and the FCAT as the fairest, most accurate, reliable and valid tool of student achievement. The apparent need for education reform in Florida was the basis for creating content standards that schools and districts are responsible for teaching for each subject taught at each grade level. The FCAT was implemented to gauge the extent to which children are learning the content standards.
If you are seeking FCAT data, you should go to the FCAT Home Page, which houses all information about the FCAT that our office creates for the public, with the exception of publications containing actual student responses that are not copyrighted for Internet publication and data spreadsheets our office is currently making ADA compliant for Internet publication in order to adhere to federal law. You will find the website located at http://www.firn.edu/doe/sas/fcat.htm, and you may find the Publications for Educators section particularly useful as you research the FCAT.
Within the Publications for Educators section of the FCAT website (http://www.firn.edu/doe/sas/fcat/fcatpub2.htm), you will find the FCAT Handbook. It provides a comprehensive explanation of the history, scoring, and reporting of the FCAT. You may find the “Scoring the Test” section particularly informative, since you are seeking information on the reliability and validity of the test. At this address, you will also find the Technical Reports for 2000 and 2002. The Technical Reports for 2003 and for 2004 are currently undergoing final review and Web tagging for ADA compliance; they will be uploaded to this location within the next few months. You may also find Lessons Learned—FCAT Sunshine State Standards and Instructional Implications Data through 2000 helpful, since it analyzes the construction phase of the FCAT.
On the Publications for Students and Parents page (http://www.firn.edu/doe/sas/fcat/fcatpub3.htm), you will find the Assessment and Accountability Briefing Book. This publication is produced every two years by the Office of Assessment and School Performance to inform the Legislature about Florida’s K-12 assessments. The publication that is currently available online was last published in 2004, and the new Briefing Book should be available in the same location sometime in February 2006, just prior to the 2006 Legislative session. The Briefing Book contains the reliability and validity data you are seeking, and it is produced by the Office of Assessment and Accountability. In the chronology of the document, you will find information regarding low performing schools and their progression. For instance, in 1995, the year prior to the adoption of the Sunshine State Standards, 158 schools (7 percent of the total number of schools reported) were identified as critically low. In 1996, the number decreased to 71, in 1997, the number decreased to 30, and in 1998, there were only four critically low performing schools. The Briefing Book also contains a breakdown of committees and educators involved in the development of the most recently administered FCAT. For the 2004-2005 administration, the Department of Education’s Test Development Center convened and facilitated 76 different committees of Florida educators and other citizens during 38 weeks in the interval from July 1, 2003 to June 30, 2004. 611 members participated in these meetings. Of the 67 counties in Florida, 63 had representatives on at least one of these committees.
After reading the materials mentioned above, I think you will see that the Department of Education has gone to great lengths to ensure that the FCAT is the fairest, most accurate, reliable, and valid tool for measuring the Sunshine State Standards. Thank you for your interest in the statewide assessment program. I hope this information is helpful to you and your wife.
Sincerely,
Cornelia S. Orr, Ph.D. Director Assessment and School Performance
**************************************************************************** (Thursday, January 5, 2006)
Hello Cornelia S Orr, Ph.D
Thank you so much for responding to my letter. Over the past year I have been on a sort of quest, spurred on by my son entering the Florida public school system. In addition to reading research on the topic of education, and the FCAT information on the FLODOE website, I have also spoken with numerous professors of Education from FSU, UF, USF, and UCF, both in person and over the phone. I have also spoken with teachers who everyday are faced with the reality of what FCAT is doing to students and curriculum.
From the research I have read and from my talks with teachers and professors alike, it appears that there is so much research out there coming from respectable sources and organizations (in my correspondences with Richard Hardy of the Curriculum Office I listed some of this research), that suggests that High-Stakes tests, such as the FCAT, and the environment which is created by such tests, are detrimental to real, emotive, meaningful learning by students.
Thus, in one hand I have massive amounts of research projects and various studies, along with the testimony of professors and classroom teachers, suggesting that FCAT-like situations cannot produce real learning, and in the other hand, after three months of attempting to have the FLDOE provide me with the name of at least one research study that refutes any of the research I have read, after three months of contact with the FLDOE, I still do not have the name of a single research study.
In your response you write:
"The Florida Commission on Education Reform and Accountability began conceptualizing the FCAT well before the first test was administered in 1998. In 1995, the Commission recommended procedures for assessing student learning in Florida that would raise educational expectations for students and help them compete for jobs in the global marketplace because seven percent, that is 158 public schools, were identified as performing at a critically low level."
My same question applies, what research studies did the members of the Florida Commission of Education Reform and Accountability provide Florida supporting the creation and implementation of the FCAT as the most reliable, accurate, fair, and valid tool of measurement. I have to hold out hope that Florida can provide me with the studies which is promised by No Child Left Behind's the "scientifically based research." I cannot understand why the FLDOE has not yet provided parents with the names of the research it uses.
And this is why we remain,
Concerned Parents
**************************************************************** January 25, 2006 Dear Mr. McCloud: Thank you for contacting the Florida Department of Education again for information about the Florida Comprehensive Assessment Test® (FCAT). I have been asked to expand on the information provided in the previous correspondence on behalf of Education Commissioner John Winn. I am sorry you feel that the requested information was not provided in the email sent to you on January 4. I attempted to provide you with the information you requested, but no actual list exists of resources “the Commission [uses] to support the FLDOE’s use of the FCAT as the fairest, most accurate, valid, and reliable measurement tool for student achievement.” Although this letter does not come to you in the form of a list, the information that follows is an attempt to provide you with the content that proves the FCAT is fair, accurate, valid, and reliable. First, let me clarify that the FCAT was developed in response to the Legislature requiring content standards and a statewide assessment that measures those content standards. If you wish to review the Florida Statutes that mandate that the Department of Education create and implement a statewide assessment program, they are located at http://www.leg.state.fl.us/Statutes/inde....II#TitleXLVIII. In your request, you specifically use the words valid and reliable. As explained in the previous correspondence, reliability and validity data are in the Assessment and Accountability Briefing Book, which is located at the following Web address: http://www.firn.edu/doe/sas/fcat/pdf/fcataabb.pdf. A more current version of this publication will be available this spring. Technical reports for 2000 and 2002, produced in conjunction with Harcourt Assessment, Inc., are currently available online at http://www.firn.edu/doe/sas/fcat/fcatpub2.htm. As explained in the previous correspondence, the technical reports for 2003, 2004, and 2005 will be available soon. If you are seeking information about the reliability and validity of the FCAT, I encourage you to review the reports that are currently available online because they explain the theoretical basis of FCAT scoring and reporting. You will find a list of references at the end of both reports. The reference lists provided at the end of these reports are the only actual lists of research I have to provide. The only additional resource listed as a reference in the technical reports not currently available is:
Baker, Frank (2001) The Basics of Item Response Theory. ERIC Clearinghouse on Assessment and Evaluation. College Park, MD: University of Maryland. Standards for Educational and Psychological Testing (1999), published and produced by the American Education Research Association, American Psychological Association, and the National Council on Measurement in Education, is a guide our office uses for FCAT administration, so I encourage you to read it as well. In addition to these resources, the Office of Assessment and School Performance subscribes to the following periodicals to stay current on issues regarding student assessment: Educational Researcher, Review of Educational Research, JEM: Journal of Educational Measurement, Educational Measurement: Issues and Practice, Florida Educational Leadership, and Educational Week. If you wish to read about how our office ensures that the examination is fair, I strongly encourage you to read the FCAT Handbook and the Assessment and Accountability Briefing Book because they discuss test development. As explained in my previous email, the Department of Education’s Test Development Center convened and facilitated 76 different committees of Florida educators and other citizens during 38 weeks in the interval from July 1, 2003 to June 30, 2004 for the 2004-2005 administration of the FCAT. 611 members participated in these meetings. Of the 67 counties in Florida, 63 had representatives on at least one of these committees. The committees involved in this process are described in the FCAT Handbook, and they include a Bias Review Committee and a Community Sensitivity Committee to ensure that the FCAT is unbiased and fair. If there is any question whatsoever as to the fairness of an item on the FCAT, the item is immediately discarded. We use these committees for this purpose due to the diversity of the state. The Department of Education is committed to ensuring that the FCAT is a fair, accurate, valid, and reliable assessment. I assure you that we conduct research to ensure the test is valid and reliable. I hope that you concur after reading the resources mentioned in this correspondence. Thank you for your interest in the statewide assessment program. Sincerely, Cornelia S. Orr, Ph.D.
The Emails (Part 2) « Result #7 on Mar 27, 2006, 8:39pm »
( Tuesday, November 1, 2005) To: Susan.Watt@fldoe.org Hello,
About a month ago I contacted this office (Richard Hardy) looking for a list of research studies the state of Florida uses to back its stance for the creation/implementation of the Sunshine State Standards and the creation/implementation of the FCAT. The research I was seeking would "prove" that state standards and a standardized state-wide test were the fairest most accurate tools to measure academic success of thousands of Florida multicultural students who use a wide variety of multiple intelligences. After providing me with no real research this office bounced me to the Office of Assessment. Well, the Office of Assessment has returned the favor. Mr. Gary Sabitsh, of said Office of Assessment and Performance, has instructed me to (re-) contact your office on my quest (which a quest is what it is turning into) for research. So, again, can this office please provide me with research, or at least give me a list of research performed or cited by any panel member that Florida may have hired to create the SSS and FCAT. I find it both scary and heartbreaking that such a basic request -the WHY of the SSS and the FCAT over any and every other tool of measurement- cannot be answered in a timely manner by the state of Florida.
Concerned, G. *************************************************************************** THERE APPEARS TO BE A MISSING EMAIL HERE (A RESPONSE FROM SUSAN WATT) *************************************************************************** (November 22, 2005) Hello Susan Watt,
Three weeks ago I sent to your office the letter reproduced below. I have yet to receive a response. Can you please explain why it is taking so incredibly long. I would think the sort of information I am seeking would be readably available to disseminate to anyone asking. However, this is not the case. It is not in any Florida Education literature, it is not on the FLDOE website, and, despite my repeated(!!) efforts, it is not available from FLDOE staff when directly contacted. Can you please tell me why? Seeing as how my son is a part of this states educational system, both he and I deserve answers.
Beyond Concerned, G.
************************************************************************** (Wednesday, November 23, 2005 )
Mr. McCloud:
The information included below may assist in your inquiry for research relating to the implementation of a standards based educational system for Florida's public schools. The developmental process of a standards based accountability system in Florida resulted from the emergence of statutory language and has been continued to support student academic achievement and the federal requirements for testing and accountability as a result of the No Child Left Behind Act. The Bureau of School Improvement of the Department of Education is responsible for reviewing and developing the Sunshine State Standards as a component of the accountability system. In an earlier communication to you, the current rationales for standards revisions in Florida were provided.
Standards-based reform (SBR): Like states around the nation, Florida is engaging in what is called standards-based reform. Because of the need for highly educated citizens in the 21st century, it is necessary to raise the rigor of academic achievement of students. SBR is a program of school improvement involving setting high standards for all students and a process for adapting instruction and assessment to make sure all students can achieve the standards (CRESST). In essence, SBR requires students to achieve high and challenging standards at determined times in their educational career that are necessary for them to move forward in their education. If students are not meeting these standards, educators must find ways for the students to learn, and this includes teachers, school administrators, district administrators, school board members, and legislators and it includes allocating the resources to do this. In addition, schools are being held accountable for students achieving these higher standards. While critics warn that it could lead to test-driven instruction that impedes implementation of thematic or interdisciplinary learning and raise questions about who should define standards and on what level. Advocates argue that standards provide a necessary foundation for educational improvement by helping focus the curriculum on what is important, raising achievement by making expectations clear to students, and providing a common language for talking about teaching and learning.
Sunshine State Standards, local implementation: The Sunshine State Standards identify the essential knowledge and skills that students should learn and for which the state will hold schools accountable. Nevertheless, how the standards and benchmarks are organized within a specific curriculum, how they are taught with what instructional materials, how much time is spent teaching them are local decisions. The state will, however, assess achievement of these standards in reading, writing, and mathematics through a statewide test.
Again, thank you for your inquiry, Richard Hardy
********************************************************************** (Thursday, November 24, 2005 )
Dear Mr. Hardy,
Once again, not a single research project was cited. You say, “Advocates argue that standards provide a necessary foundation for educational improvement by helping focus the curriculum on what is important, raising achievement by making expectations clear to students, and providing a common language for talking about teaching and learning.” What research do these “advocates” use? Can I please just have that information without all the run around? I don’t mean to sound pushy, but I’m extremely frustrated at the FLDOE’s refusal to give this information out.
I have read through the Sunshine State Standards, how exactly does the FCAT test a majority of the English standards? The Standards and Benchmarks for viewing, listening, speaking, and literature for instance? I have noticed that only 30% of the FCAT reading selections are pieces of literature for the 10th grade test (the other 70% being informational pieces), what research does your office have to suggest that today’s students no longer need the higher-order critical thinking skills necessary to construct meaningful, individual interpretations of literature, opting instead to force the “importance” of informational pieces? Research seems to suggest the former takes more critical thinking skills.
Below are some research projects (the tip of the iceberg, sort to speak) which seem to contradict the “advocates” of standards-based reforms:
The research of Richard Allington demonstrates the distortions and lack of scientific evidence in the National Reading Panel’s Report which said report was embraced by both the Federal government and various State governments. Said report was used as a rational to implement certain changes in Standards Bases Reform.
Janet Allen’s (EdD) research with Literacy Workshops suggests that skills learned in the workshop arena may not be reliably measured by a “fill-in-the-bubble” standardized test.
The research of Kathy Emery and Susan Ohanian demonstrates the Business Roundtable’s unhealthy involvement with numerous state’s departments of education.
The research of James Paul Gee and Maryann thingyar suggests that because of the huge difference in the acquisition of various primary discourses a large segment of Florida’s student population is already a step or two behind their classmates, whose primary discourse more closely matches that of the language used in the FCAT. It seems that a one-size-fits-all set of standards and a one-size-fits-all standardized test would unfairly benefit some students over others.
Gunther Kress’ research into the use of genre in education suggests that, just as students have to mold their self expression into nice, neat categories, that in reality do not exist, students have to mold and bend any epistemological urge and conception to match what the State deems appropriate rather than that of the most immediate expert, namely the teacher.
The research of Gordon M. Pradl and John S. Mayher suggests that contrary to their stated purpose of promoting high order thinking skills, rigorous state testing programs actually narrow the parameters of critical thinking in the student population.
The research of Kathleen and James Strickland suggests that the behaviorist approach of Bloom’s Taxonomy may be lacking in value for today’s educational demand. Rather, educators should assist students in their own pursuits to actively create meaning from elements in their environment. This would necessitate choice in what is learned and how it is learned, and of course how the learning is to be assessed. And I doubt a majority of students (or a majority of adults) would select a fill-in-the-bubble standardized test as the means to which they will demonstrate their leaning. When was the last time you took a fill-in-the-bubble test to prove that you are competent in the skill that you should possess?
The research of Judith A. Langer demonstrate the inability of a standardized test to measure true meaningful knowledge, the kind that schools should be promoting.
The Emails (Part 1) « Result #8 on Mar 27, 2006, 8:36pm »
The following series of email exchanges begins with a request put to the Florida Department of Education for research proving that the FCAT was the MOST fair, valid and reliable tool of student measurement, as was claimed by Commissioner Winn in a recent email disseminated to teachers statewide (see below). Unfortunately, the original email request has been lost. Thus, the exchange begins with FLDOE’s reponse to that request. Commissioner Winn states (in regard to performance pay): “Student learning gains must be a ‘primary’ component in the evaluation of outstanding teachers. (The rule identifies the FCAT as the most reliable and valid measure of student learning gains in reading, mathematics, and eventually science.”
From: Hardy, Richard To: lucy****@****.net Sent: Wednesday, September 28, 2005 2:22 PM Subject: request for information from FLDOE
Mrs. G*****:
Please allow me to attempt to respond to your inquiry relating to standards based curriculum and testing in Florida’s schools. Mrs. Watt is not available this week and requested that I may be of assistance.
I will respond to each of your inquiries in the order they were proposed.
1. The conception of state wide grade level expectations, standards, standard classroom curriculum and statewide assessments emerged as a response to the educational, political and economic interests focusing on the competency levels of students nationwide. The development of the current grade level expectations and standards were developed with input from school district stakeholders in education and intended to compliment the development of curriculum for students. In a review of the standards, you will note a lack of specificity in verbs and broader terms such as knows or understands. The intent was to allow district and school level curriculum developers to have flexibility to enhance the standards to best benefit student’s academic skills and knowledge. The standards movement continued to gain recognition and with the advent of state wide testing, the opposite sometimes occurred. Education officials, politicians and citizen groups required the proposal and measurement of specific curriculum through state testing. Secondly, with the expediential growth that Florida has experienced in the last decade leading to intense teacher recruitment and the multicultural experience expanding into many classrooms, the standards became a focal point for many instructors and administrators. Again, rather than being a guideline for curriculum, the standards emerged as the centerpiece. Florida’s standards have undergone several external reviews, first from the Council of Basic Education and more recently reviews from the AP College Board and Dr. Will Daggett’s International Center for Leadership in Education. All three agencies commented on the breadth of the standards and the need to spiral up or build complexity from grades 6-12. Overall, the current standards and grade level expectations were cited as reflecting a commonality with other states and national standards.
2. The Florida Comprehensive Assessment Test is intended to measure the academic achievement of all students in the state of Florida. Research conducted from the Department of Education may only reveal how a student may progress or regress in content knowledge from year to year and at what level of competency they hold regarding a particular standard based curriculum. You may want to contact the Office of Assessment and Testing at the Department of Education for additional research in this arena.
3. I would be hesitant to challenge the need for students to not be aware or examine concepts such as the main idea of written passages. Research does reveal that a significant number of students in Florida’s schools struggle with effective reading comprehension and the interpretation of text and other reading sources. Requiring students to analyze and evaluate reading content for literal and interpretive meanings seems justifiable.
4. Referencing your opening comments reflecting on the discrepancy between university level discussion of pedagogy, teaching method and the real world application of classroom best practices, I would concur. Colleges of Education focus on the method; the classroom teacher is responsible for the examination of content and curriculum. This practice takes time, patience and on the job development. Many teachers must learn content while teaching, and are not offered significant professional development or compensated for an extension of reviewing teaching methodology.
Again, thank you for your inquiry and if I may be of further assistance please contact me.
Sincerely,
Richard Hardy Director of Curriculum Support Bureau of School Improvement (850)-245-0764 richard.hardy@fldoe.org
************************************************************************ -----Original Message----- From::lucy****@****.net] Sent: Wednesday, September 28, 2005 8:51 PM To: Hardy, Richard Subject: Re: request for information from FLDOE
Mr. Hardy,
Thank you for your timely response. And to be honest, aside from the fact of my son's entrance into Florida public schooling, a lot of the motivation for this going-on-a-year-now investigation stems from my personal dissatisfaction with my high school experience o so many years ago. Even though my mother was a 6th grade teacher back in Illinois, I almost did not make it through high school. In addition, my wife dropped out of high school her junior year for reasons that I hear echoed in so much of what I have been reading about Education.
Points:
1. Thank you for the references to the external reviews by the Council of Basic Education, the AP College Board and the International Center for Leadership in Education. I would ask, are there any more? I hope to find the Research within these institutions which backs the claim that State selected and imposed standards benefits students over other options.
2. Thank you for the reference to the Office of Assessment and Testing at the Department of Education.
3. While I agree with you that interpretative and comprehensive ability is important, how can a single test (the FCAT) hope to ever fairly and accurately describe the levels of ability for such an incredibly diverse and multicultural student population. Again, the research I have done and the talks I have had seem to strongly indicate that this test cannot do that. What research does Florida have that says differently? Should this question be posed to the Office of Assessment and Testing?
Thank you again for your time and patience. I am very moved by all of this. It is such an incredibly fascinating journey into Education that I have been on. I really had no idea of the battles and massively different stances that have been discussed. As well, the intriguing ideas: I'm reading about something called multiple intelligences at the moment. Very neat.
Following the Research, Concerned ********************************************************************** From: Hardy, Richard To: lucy****@****.net Sent: Friday, September 30, 2005 9:44 AM Subject: RE: request for information from FLDOE
Hi A:
I am glad you could use the responses I sent and I agree that the complexity of educational philosophy, practices and choices is both interesting and perplexing.
I like your question relating to the standards based tests we currently have across the nation. I would concur that we have yet to develop a test or set of tests that can provide educational personnel enough relevant information on student learning. The current testing philosophy proposes that we may be able to identify the greatest gaps in student learning for those areas tested. Secondly, the testing mandate is strongly recognized at the national and state political level, so the charge given to the Florida Department of Education is to assure that we are assisting districts, schools and students with both the relevant curriculum and appropriate assessments. This becomes quite an interesting puzzle as we review data that supports that many students are very successful with the FCAT test, while others seem to genuinely struggle with the test.
The goal is to be able to provide all students with the knowledge and skills to be proficient, independent and lifelong learners. We at DOE have daily discussions looking for a better way to do this as do many teachers, administrators, parents, college professors, ed consultants and students. In the Bureau of School Improvement we celebrate the gains of students and agonize over the students who still struggle. Hopefully, testing will diminish as student skills increase and educational practices evolve.
I think you will find the research of the International Center for Leadership in Education a valuable resource. Dr. Daggett has produced a wealth of information regarding current trends, the rigor and relevance of curriculum and testing and future concerns for public education in America in the global arena. Again, thank you for your interest and best of luck.
*************************************************************** From: z****@****.com Sent: Friday, October 07, 2005 11:13 PM To: Hardy, Richard Subject: FCAT Question
Dear Mr. Hardy,
My wife and I have been in recent contact with you regarding some questions we had about the Florida Department of Education's stance on the use and purpose of FCAT and state-wide standards. We are still looking for answers as to what resources Florida is using to back the above mentioned.
In a recent email you stated, "The current testing philosophy proposes that we may be able to identify the greatest gaps in student learning for those areas tested." This, to us, brings up two concerns:
1. This may be only a semantics issue, but your statement seems to present a notion that the FCAT focuses on the skills it tests, skills dubbed "essential," according to the Pinellas County school system, merely because these are the only skills that this test may be able to measure(?).
and,
2. If the case is being made that the FCAT is able to test these skills in a fair and accurate manner, I ask what "current testing philosophy"? In an earlier email you mentioned three sources of "external reviews." the AP College Board, the Council of Basic Education, and William Daggett's International Center for Leadership in Education. The Council of Basic Education no longer exists. And it seems that William Daggett is constantly surrounded by controversy due to the way he misrepresents and misinterprets data , and may go so far as to flat-out lie in his speeches. I am making reference to information found at the following websites:
What is the Florida Department of Education's official stance on the controversy surrounding Dr. Daggett, a man who appears to be very influential in Florida Education?
You also stated, "I think you will find the research of the International Center for Leadership in Education a valuable resource. Dr. Daggett has produced a wealth of information regarding current trends, the rigor and relevance of curriculum and testing and future concerns for public education in America in the global arena." You have referred me twice to Dr. Daggett. I was unable to find any research on the website of Dr. Daggett's International Center. Most links on the website ended in either a page full of quick factoids full of buzzwords and other over-generalized jargon of education, or a page attempting to sell me some package of Dr. Daggett's. I wonder at the motivation to not readably disclose information about the Center's research on its own website.
All my wife and I are asking for is a list of research studies that the state possesses to back its stance. And while I understand that Florida may have gathered together, at various moments in time, an assortment of people involved with education, surely these individuals had to provide Florida with their research, in order for your state to go ahead with the implementation of the issues we are concerned with, namely the FCAT and state-wide standards. What was this research?
As a final note, on rereading this email, I find that it sounds a bit...antagonistic. I apologize. The antagonism is not intended. We are frustrated by the still large discrepancies we are finding.
Concerned, G
************************************************************* (Monday, October 10, 2005) Mr. G:
Thank you for your communication and insight regarding the standards based component of education if Florida.
I have provided you with the information the curriculum office has available to offer at this time regarding the development and review of the Sunshine State Standards. Our office does not focus on the assessment of the standards. I have included the website for the Assessment office if you would like to contact them.
*************************************************************************** (Tuesday, October 25, 2005)
To: FLDOE Assessment Office
Hello,
This is my third attempt to contact the Assessment Office, with no reply in return. I am searching for research the state of Florida has used in originating and evaluating the Sunshine State Standards as well the FCAT.
G__
*************************************************************************** (Wednesday, October 26, 2005) (Notice how the message is bounced around the DOE)
G__,
Please see message below, I hope the folks in Curriculum Support are able to assist. There is also a technical report for FCAT located at:
Gary Sabitsch, Supervisor K-12 Assessment Reporting Office of Assessment and School Performance
-----Original Message----- From: Kent, Judy Sent: Wednesday, October 26, 2005 9:33 AM To: Sabitsch, Gary Subject: RE: Third Attempt to Gather Information
Gary,
I don't have what he is asking for. The FCAT is based on the Sunshine State Standards. For the Performance Tasks and Item Specifications we use (which are based on the SSS), send him to http://www.firn.edu/doe/sas/fcat/fcatis01.htm. If he asks, we are holding off on releasing the new Specs. for Reading and Science because all SSS are currently under revision and will change after the legislative session.
If he needs to get in contact with someone with more information on the SSS, he needs to contact Curriculum Support: their hotline is 1-800-471-8276 and email is currsvcs@fldoe.org.
Judy T. Kent Information Dissemination and Reporting Assessment and School Performance Florida Department of Education
-----Original Message----- From: Sabitsch, Gary Sent: Wednesday, October 26, 2005 9:15 AM To: Kent, Judy Subject: FW: Third Attempt to Gather Information
Judy, do you have anything that can help this person?
Gary Sabitsch, Supervisor K-12 Assessment Reporting Office of Assessment and School Performance
Introduction to the Emails (Read First) « Result #9 on Mar 27, 2006, 8:11pm »
What is the MOST fair, reliable and valid approach to assessing student learning?
Hello,
The following series of email exchanges (still on going as of March 31, 2006) between myself ( and my wife) and the Florida Department of Education began as a simple request on my part for the research the FLDOE had to back its stance that the FCAT is the MOST fair, reliable, and valid tool of measuring real, meaningful learning on the part of Florida's public school students. In a recent email sent to all Florida teachers, Commissioner Winn categorically stated that the FCAT is the MOST reliable and valid tool of measurement. Mr. Winn's statement, as you will find, cannot be backed up by any research that the FLDOE has performed or possesses. The full text of Mr. Winn's letter can be found in another discussion board on this forum.
At this point I would like to present a hypothetical situation, one which I posed to Dr. Cornelia Orr of the FLDOE, and can be found in this series of emails:
If one wished to get to the corner store for the daily paper how would one do it? Walk? Well, that is certainly a valid and reliable way. Bike? That too seems valid and reliable. Drive? Take the bus? Skateboard? Scooter it? Skip? Jog? Sprint? Rollerblade? Etc. There are many approaches. Let us say in our example, and I apologize if it is a bit simple, but: let us say that the standardized test approach is walking. Or it could be the skateboard. Or the car. It doesn't matter, because if we wanted to find out the M O S T valid and reliable way we would have to compare ALL the approaches, we would have to research all the approaches to determine the most valid and reliable. Again, I apologize for the basic comparison, but I think (hope) I'm getting my point across. A comparison of all approaches seems like the first step in coming up with any solution...
In my mind, as I just stated, the very first step would be to compare approaches, for how does one know what the best approach is if one doesn't look at all options. As you will find in this series of emails, the Florida Department of Education has not done this comparison of approaches. It has, for various reasons, decided upon one approach, the standardized test approach, the FCAT, an approach that has been shown time and time again to be dangerous to students' physical health, mental well-being, and a deterrent of any real, personal, meaningful learning.
A note about the emails. They come in four parts. Within each part, the emails are separated by a line of asterisks. I have tried to keep the original text of the emails as authentic as possible. I have left unedited any grammatical issues, as well usage issues. However, from time to time I have changed the names and emails of some who could be retaliated against for their involvement in this quest. Also, some emails are missing. When it is appropriate, I will mention when an email has disappeared. Please be warned, I am not an expert of education, nor literature, though I have studied both. Above all, what I am, is the parent of a child enrolled in the public school system, and this fact was the genesis of my and my wife's request for research.
I hope you find something useful in these emails. If you care to respond to them, pro or con, feel free, after all this is a forum.
![[Home]](http://s1.images.proboards.com/menu/home.gif)
![[Help]](http://s1.images.proboards.com/menu/help.gif)
![[Search]](http://s1.images.proboards.com/menu/search.gif)
![[Login]](http://s1.images.proboards.com/menu/login.gif)
![[Register]](http://s1.images.proboards.com/menu/register.gif)
![[Search This Thread]](http://s1.images.proboards.com/search2.gif)
![[Reply]](http://s1.images.proboards.com/buttons/reply.gif)
![[Send Topic To Friend]](http://s1.images.proboards.com/buttons/sendtopic.gif)
![[Print]](http://s1.images.proboards.com/buttons/print.gif)
Author
![[Quote]](http://s1.images.proboards.com/buttons/quote.gif)
Logged
Author