2c.1. In what ways does the unit regularly and systematically use data to evaluate the efficacy of and initiate changes to its courses, programs, and clinical experiences?
The unit regularly and systematically uses data studied during Data Decision Day (D3) and information reviewed during meetings with support committees to evaluate the efficacy of and make changes in courses, programs, and clinical experiences.
Department of Education faculty meet for annual Data Decision Day (D3) sessions to review data generated by candidate and program assessments. Often these sessions are held at an off-site from the unit, but still remain on the university’s campus. Faculty subgroups analyze data, formulate conclusions, and propose recommendations for action. For example, in fall 2008, in baccalaureate initial licensing program, candidate portfolios would require Portfolio Essays, rather than artifact rationale statements. The change was based on the faculty’s review of the portfolio process and related data, concluding that rationale statements did not generate data that supported conceptual framework performance outcomes. The History of Change document, 2003-Present, chronicles actions taken in response to data-based evidence and other factors.
In 2008-09, faculty began targeting scrutinize specific areas of the conceptual framework, rather than attempt to assess all areas. For example, the targeted review began with analysis of two of the six areas of the framework, “Learner” and “Pedagogy.” Adhering to the assessment cycle process ensures that focus is maintained on all areas of the Conceptual Framework annually, and that the complete cycle is addressed every three years. The Conceptual Framework Assessment Cycle outlines the target areas and related requirements. In 2009, the unit established the UAS Review Plan, which establishes a chronology and responsibilities for reviewing and recommending changes to components of the assessment system.
Several examples of regular and systematic use of data to evaluate the efficacy of and initiate changes to its courses come from the following committees: (1) Teacher Education Committee (TEC) meets monthly. TEC receives data on and feedback regarding courses, programs, and policies affecting the Department of Education and departments that support secondary teacher education; (2) the Teacher Education Advisory Council (TEAC), which meets once per semester. It shares information with partners and makes recommendations regarding program and department matters; (3) the School Counseling Advisory Council meets at least once per semester. Information on initiatives is shared and input is requested for planning and data review. Recommendations are used for program development and improvement; (4) The Exceptional Needs Advisory Council meets once each semester. Data are shared regarding the major program revisions completed in fall 2009.
The committees’ minutes are available in the on-site Exhibit Room.
2c.2. What data-driven changes have occurred over the past three years?
Several data-driven changes have occurred during the past three years as related to candidate performance, program quality, and unit operations.
Examples of data-driven changes related to candidate performance follow. In response to two concerns regarding candidates’ Formal Lesson Plan performance, in fall 2009, the unit’s Department of Education incorporated the Lesson Planning Guide (LPG) into practica courses and student teaching. The LPG is designed to provide structure and guidance to candidates as they plan lessons. This change is based on data from candidate performance on Formal Lesson Plans. The first concerns were challenges in content knowledge criteria on field-based signature assignments. Candidates found required assistance in areas of differentiated instruction and student adaptations and modifications on Formal Lesson Plans. Also, introduction of the LPG was prompted by evidence of challenges candidates faced in considering students’ prior experiences and knowledge in order to adapt and differentiate lessons. For example, on the Formal Lesson Plan, secondary education / mild intervention candidates earn mean scores ranging from 2.53 to 3.04/4.00.
To improve scores in the area of pedagogical content knowledge of post-baccalaureate initial licensing mild intervention candidates, instructors have increased requirements regarding the use of evidence-based practices in signature assignments. Three examples of signature assignments that must include evidence-based practices are: the Strategies Notebook in Methods, SPED 508/509. The Family-School Partnership Action Plan and Reflection Notebook assignments in Collaboration and Communication in Exceptional Needs, SPED 537 require documentation of research-based practices. Also, lesson plan submissions in methods courses, practica, and student teaching require documentation of evidence-based strategies, materials and methodologies. Finally, the project proposal in Teacher Inquiry, SPED 550 requires evidence of the research base in the literature review and design phases.
There are several data-driven changes related to program quality. Alumni Survey responses in the neutral to negative range appear on one item associated with pedagogical content and skills: knowledge of resources that could contribute to professional development. In response, the unit’s Department of Education has infused Educational Psychology and Measurement (EDUC 250), Advanced Practicum (EDUC 310), and methods courses with assignments that require candidates to access and utilize teaching resources from both electronic and print sources. As well, the unit has scheduled professional development topics and guest speakers who address assessment and instruction topics such as, high ability gifted and talented education, second language learning, and current issues in special education.
In Undergraduate Exit Survey responses, candidates have requested additional information on the following topics: legal rights and responsibilities of teachers, learning theory (Pedagogy), how to implement various approaches of classroom management, and resources (association, journals, etc.). In addition to addressing these topics through student teaching seminars, instructors have responded by increasing instructional time devoted to the topics in courses
Another program change has resulted from a need not met by data from an existing assessment instrument. In the baccalaureate initial licensing programs, at times, faculty have voiced concerns regarding candidates’ commitment to their programs of study. A source of information that addresses commitment, the Assessment of Professional Dispositions instrument, provides data primarily of a summative, holistic nature. Faculty wished to formally remediate candidate dispositions in a timely manner. The result was the institution of the Candidate Advancement Committee (CAC). In the CAC several faculty review performance information with the candidate in question and establish a remediation plan that sets short-term and intermediate-term benchmarks.
In 2006-07, data from the baccalaureate initial licensure dual programs’ Undergraduate Exit Survey indicated that program graduates wish for more information on resources for exceptionalities, legal issues, and instructional technology. Additionally, data from the Employer Survey indicated a need for greater attention to preparing candidates in two areas: Work with high ability gifted and talented students, and integration of technology into instructional processes. Consequently, the topic of high ability gifted students was added to the student teaching seminar. To address technology integration into instruction, Technology Applications in Teaching, EDUC 205, was revised to make technology integration the focus of the signature assignment. Also, technology integration became an assessed criterion in the signature assignment lessons that candidates planned and implemented in their field placements.
Post-baccalaureate programs’ Employer Survey data indicated a need to increase candidate skills at integrating technology into instruction. In response, the signature assignment in Technology Applications in Teaching, EDUC 502, was revised with a focus on technology integration. As well, a technology integration criterion was added to field placement evaluations.
Data from the Employer Survey prompted changes to better address issues of collaboration with educational personnel. In Communication and Collaboration in Exceptional Needs, SPED 537, the School Action Plan signature assignment specifies details for establishing and fostering partnerships and collaboration with parents to improve student learning. An additional data-driven change from the Employer Surveys was development of signature assignments to address the formulation of plans focusing on candidates’ professional development related to curriculum and assistive technology in the field of intense intervention.
Several changes in the baccalaureate initial licensure clinical experiences have resulted from data provided by candidates on the Undergraduate Exit Survey and by cooperating teachers on the Field-Based Experience Assessment Survey. The first change involved lengthening the student teaching experience from 15 to 16 weeks. A second student teaching change entailed decoupling knowledge and skill assessment from dispositional assessment. The latter change resulted in two distinct evaluation instruments to assess student teachers, the Student Teaching Evaluation instrument and the Assessment of Professional Dispositions instrument. A third change resulted in significant reduction in the number of indicators comprising the knowledge and skill-based Student Teaching Evaluation instrument (2006 - 2008) (2009).
Data-driven changes have also occurred in regard to unit operations. Data from the Noel-Levitz Student Satisfaction Survey affirmed faculty’s perception of the need to change the single advisor model. Originally, secondary education candidates’ advisors were faculty exclusively from either a subject discipline or the Department of Education. In 2005, the dual advising model was implemented, providing candidates with advisors both in their subject disciplines and education. In school counseling, based on the Employer Survey, all field evaluation instruments have been updated. Due to the lack of useful information generated by the original formats, evaluation instruments were redesigned to capture greater variability in candidate performance levels and to more realistically reflect expectations placed on school counseling candidates in their field placements.
The unit’s History of Change documents all major changes in the unit’s programs and operations for the past seven years. Many changes are data-driven.
2c.3. What access do faculty members have to candidate assessment data and/or data systems?
Faculty members have access to several candidate assessment data and data systems. These include Jenzabar, Cougar Connection, Candidate Tracking System (CTS), TaskStream, and Blackboard. Faculty have secure individual access codes to confidential information stored on these systems.
Candidate transcripts are accessible to faculty through the university’s student management system, Jenzabar. Faculty have direct access candidates’ Praxis I scores via the web portal, Cougar Connection, at its Pepin link. Through the CTS, faculty may directly access information candidates’ transition point status. The CTS, however, is used most often by faculty for information when making decisions on candidates’ admission to teacher education programs and other transition-point specific decisions.
TaskStream is the unit’s assessment data system used by faculty to assess signature assignments. TaskStream allows faculty direct access to assessment data on candidates currently enrolled in the faculty member’s courses. To access candidate assessment data from past courses, faculty enlist assistance from the unit assessment system manager. Faculty may elect to utilize Blackboard to manage course grades and attendance. Finally, faculty have direct access to candidates’ hard copy files.
2c.4. How are assessment data shared with candidates, faculty, and other stakeholders to help them reflect on and improve their performance and programs?
Assessment data are shared with candidates by means their access to data systems and through communications with faculty. Candidates access assessment data about signature assignment performance on TaskStream. These data include numerical scores and faculty comments. Candidates may access their transcripts and Praxis I scores through Cougar Connection. Unit candidates may meet with faculty both during and outside of office hours. Candidates regularly meet with faculty to review details of major assessments and reflect on remediation requirements.
Assessment data are shared with faculty through direct involvement with data analysis, reports from unit personnel, and communication with department chairs. Faculty are directly engaged in the analysis and summarization of data results. Annually, education faculty analyze candidate and program performance data during Data Decision Day (D3). During D3, the primary focus of D3 is analyzing and reporting data on candidates’ signature assignments. Depending on availability on D3, survey data from alumni, employers, and student teachers may also be analyzed.
At department meetings, Title II Report data are shared with faculty by the unit’s licensing officer. Information such as pass rates on Praxis II exams, enrollment figures, and demographics are found in Title II Reports. Also, at department meetings, the Director of Field experiences shares assessment data from surveys completed by student teachers and cooperating teachers. Faculty’s direct involvement in data analysis and reports from unit personnel are the basis for both formal agenda items and informal faculty discussions throughout the academic year. Finally, the unit and university community have access to Noel-Levitz Student Satisfaction Survey data on advising and academic programs on the university’s intranet site. These data are also reviewed at department meetings.
On an individual basis, department chairs share data every semester with faculty based on the Individual Development and Educational Assessment (IDEA) teaching and course evaluations. These data constitute a component of faculty members’ annual performance evaluation by department chairs.
Assessment data are shared with stakeholders through formal meetings and website postings. The Teacher Education Committee (TEC), composed of representatives from secondary education programs, meets on a monthly basis. Members receive reports on issues that potentially affect the program decisions. Also, data from the unit’s Department of Education is accessible to faculty. For instance, shared data has influenced program decisions in the Department of Health and Physical Education. The department’s concern for candidates’ Praxis II performance led to the addition of one course, Human Anatomy and Physiology, BIO 222, with a human anatomy course, Human Anatomy, BIO 218. The unit’s three advisory councils meet on a semester basis to share information on program performance that may influence program decisions. For example, the School Counseling Advisory Council reviewed data on candidate field performance that led to development and introduction of the Field Placement Activities Checklist.