STANDARD 2:  ASSESSMENT SYSTEM AND UNIT EVALUATION

ASSESSMENT SYSTEM
STAKEHOLDER INVOLVEMENT
INCORPORATION OF STANDARDS IN THE CONCEPTUAL FRAMEWORK
THE UNIT ASSESSMENT SYSTEM FOR INDIVIDUAL CANDIDATES
DATA COLLECTION, ANALYSIS, AND EVALUATION
USE OF DATA FOR PROGRAM IMPROVEMENT

Assessment System
The University of Saint Francis, Department of Education program and Unit Assessment System (UAS) assists in determining if candidates/graduates are meeting the expectations set forth by the Department Theme, “Educators facilitating and advancing learning in a diverse, ever-changing society.” In order to reach this goal and enable statistically-supported decision making, the Department designed an assessment system that analyzes internal and external data. This system monitors and assess: 1) candidates’ progression through their selected programs of study, 2) individual candidates, 3) a specific program of candidates, 4) all Teacher Education candidates’ performance by decision point(s), 5) all Teacher Education candidates’ performance by indicators within decision points, 6) all Teacher Education candidates’ performance at program completion. Finally, the UAS will conduct follow-up analysis of all Teacher Education candidates’ perceptions of the Teacher Education program in relation to successful teaching performance.

Prior to the formal Unit Assessment System, the Department used separate databases for each set of data (e.g., Candidates Accepted into Teacher Education, Praxis Score Results, and Practicum & Student Teaching Evaluation Composites) to collect, monitor, and analyze candidate performance and make decisions for candidates and programs in Teacher Education. The results were then transferred into the History of Change within the Department of Education document (Table 2.5). While this system did provide the Department with a focused ability to aggregate and analyze data and decisions related to program development, it did not allow aggregated analysis by program or Department; nor did it provide analysis of multiple decision points and comparisons to be made between programs or candidates. Finally, decision-making for incidents of change in the early system was not systematic. The major barrier was the inadequacy of the separate databases that were incapable of systematically capturing and updating the necessary information (significant data existing in the University’s registration module) for the Department to methodically make decisions.

The University of Saint Francis’ Conceptual Framework incorporated a shared vision consistent with the institution’s Mission, which was aligned with IPSB, INTASC, and NCATE Standards. Since the 1997 visit, the Department with its stakeholders reviewed and revised the Framework (Appendix A, Table CF.1) to more clearly articulate the vision for preparing effective teachers for the 21st century. The NCATE 2000 Standards provided the opportunity for the development and implementation of an assessment system and the refinement of the Conceptual Framework.
 

Stakeholder Involvement
The Department of Education meets on a weekly basis. Meetings are scheduled at different times during the month to encourage participation of all members, especially adjunct faculty. The Department faculty members participated in the initial design of the Unit Assessment System. In addition to the Department meetings, the Department of Education determined that two existing committees, the Teacher Education Committee (TEC) and the Teacher Education Advisory Council (TEAC) were effective formal structures to provide stakeholder input for the assessment system.

The Teacher Education Committee consists of Professional Education faculty (Department and Arts & Science [content] faculty), and candidates representing each Teacher Education program. In the past, two faculty representatives were elected by Faculty Forum to serve as members of the Teacher Education Committee. Last year, the Department requested that Forum change the membership to include a member from each Arts & Science Department in which there were Teacher Education majors. The membership change was approved and five new members from the Departments of Arts & Science attended their first meeting in January 2002.
The responsibilities of TEC relevant to the UAS are to:

• Admit candidates to the Teacher Education program.
• Initiate and consider recommendations for policy and program changes in the Teacher Education Program.
• Coordinate the policies of the Teacher Education program with the academic departments of the University.
• Act as a review and hearing committee for candidates with regard to their admission and retention into the Teacher Education Program.

The Teacher Education Advisory Council (TEAC) includes professional education, P-12 school faculty and administrators, related education stakeholders (e.g., Director of the local teacher union, Director of the Local Education Fund), and program alumni. The following Table (2.1) illustrates that the Department includes a broad range of community professionals.

Teacher Education Advisory Council Members 2002

Table 2.1

In relation to the UAS, it is the responsibility of the Advisory Council to:

• Provide recommendations on academic and administrative matters impacting programs and policies relating to the Department.
• Connect the Department and its programs with appropriate education resources for the purpose of establishing strong public relations, program awareness and development.
• Help build a network of experienced practicing professionals to provide assistance, guidance and expertise.
• Continuously review the major offerings and programs to enhance the work of the Department.

Since these are established structures whose primary purpose is to provide input relative to the Teacher Education program, these stakeholders were logical groups to involve in the development of the UAS.  Both the TEC Committee and TEA Council are included in the procedures for the UAS to assure continued input. The focus of work and resulting recommendations in the 2002 academic year for these stakeholders has revolved around the implementation of the UAS and the effectiveness of the decision points in predicting successful candidate outcomes. The Department of Education presented to the TEC and TEAC the History of Change within the Department of Education (Table 2.5) as an early UAS. This assessment effort provided direction for accuracy, fairness, and consistency through which decisions regarding changes in practice and performance were made in the Department.

In addition to the above stated responsibilities, examples of the following activities demonstrate stakeholder involvement (TEC and TEAC) in the developmental activities associated with UAS and the Department:

• Reviewed initial design of the UAS.
• Discussed confidentiality and security issues as related to the UAS.
• Reviewed Curriculum Design Survey.
• Reviewed Critical Decision Points for Teacher Education (Appendix D, Table 2.2a & b).
• Reviewed proposed licensure framework and curriculum for Teacher Education.
• Reviewed marketing research conducted as related to the proposed Teacher Education programs.
• Discussed the Transition to Teaching opportunity.
• Reviewed final UAS – Phase I implementation.
• Reviewed the revised Teacher Education Program: Candidate Exit Survey.

Finally, the Teacher Education Committee met throughout the Summer of 2002 to refine the standards’ alignment of curriculum, program advising, and faculty monitoring of candidate progress and program effectiveness.

Evidence of the involvement stated above and recommended changes from stakeholder input are found in available documents. TEC and TEAC Minutes of all committee deliberations are kept on file and are available. All changes that occur in Teacher Education (initial and advanced) are documented in the History of Change within the Department of Education (Table 2.5) – Volume I (1987-1997) and Volume II (1997-2003).
 

Incorporation of Standards in the Conceptual Framework
The Department’s incorporation of the INTASC Principles and IPSB Standards began with the broader University and Department Mission statements. The Teacher Education Theme, “Educators as facilitators and organizers for effective decision making,” was established in 1994. Once established, the Conceptual Framework was developed to include six learning strands deemed important to the art and science of teaching.  The Conceptual Framework underwent its final revision in Summer 2002. The Department Theme is now “Educators facilitating and advancing learning in a diverse, ever-changing society”. The Conceptual Framework is aligned with the Interstate New Teacher Assessment and Support Consortium (INTASC) Standards (initial level) and National Board for Professional Teachers Standards (NBPTS) (advanced level) as is evidenced in the Teacher Education Handbooks (undergraduate and graduate) and the requirements and format prescribed for the portfolio. Since the Conceptual Framework is aligned with INTASC and NBPTS Standards and are embedded in the IPSB Standards, the Department’s programs address all standards.  Professional education members, which includes Arts & Science faculty members finalized course syllabi templates that include Conceptual Framework skills, content, and developmental standards addressed in each course (Appendix E, Course Linkage Charts). Also, faculty members mapped courses (Appendix F, Program Alignment Matrix) for each program (Elementary/Mild Intervention, Secondary/Mild Intervention, All-Grade Exceptional Needs/Mild Intervention, and Secondary and All-Grade Visual Arts) to provide a systematic mechanism that assures programs meet standards of teacher competency at the initial and advanced levels. Finally, the Critical Decision Points for Teacher Education (CDP, Appendix D, Table 2.2a & b) illustrates the connection of each decision point to the Conceptual Framework, INTASC Principles and NBPTS (graduate candidates only).

The electronic UAS database is part of the formal mechanism used to demonstrate and monitor candidate and program standards. The database is designed to collect, organize and aggregate data necessary to demonstrate program accountability. The UAS is capable of generating the Candidate Program Alignment report of those skills obtained upon the successful completion of a course, a decision point, and program.

Faculty members continue to review and monitor the alignment of all standards within specific courses and programs through a circular process of identification, implementation, and on-going assessment as follows:

• Selecting the performances/skills within the Conceptual Framework that support the IPSB Standards and INTASC Principles (identification)
• Aligning course content with the Conceptual Framework, IPSB Standards, and field experiences with INTASC Principles (identification & implementation)
• Developing and using rubrics for field experiences and all levels of the portfolio (implementation & assessment)
• Monitoring candidate progression through the decision points and providing feedback to candidates during advising (implementation & assessment)
• Reviewing program alignment matrices used to correlate IPSB Standards and INTASC Principles addressed in all Teacher Education programs (UAS reports–analysis /assessment)

In the Spring of 2002, the Department began a collaborative venture with the Indiana Department of Education, Division of Special Education Project entitled, “Individual Curriculum and Assessment Notebook” (ICAN). The purpose of this effort is to use the ICAN software to assist colleges/universities with the alignment of the IPSB Developmental and Content Standards. The ICAN collaborative effort will provide a proven and consistent method (based on the ICAN Indiana Academic Standards alignment in existence and implemented in many Indiana P-12 schools) accessible throughout the state to support IPSB Content and Developmental Standards alignment. When the alignment process is complete, the Department will use the ICAN alignment to support its program alignment with standards.
 

The Unit Assessment System for Individual Candidates
All candidates are informed of the standards in the professional education courses through discussion, syllabi, advising sessions, the Teacher Education Handbooks undergraduate and graduate), Critical Decision Points in Teacher Education (Appendix D, Table 2.2a & b), the Portfolio Manual, and the Manual for Field Experiences and Student Teaching.  The handbooks and manuals were revised in 2001 to reflect changes and better clarify the assessment system. Each course within specific programs targets standards and performances and matches those performances with products (assessment methods). The course format provides an expectancy of skills to be taught by faculty and acquired by candidates. In the introductory core courses, faculty discuss the relationship between course, program, state, and professional expectancies for teachers. Evidence of these expectancies must be demonstrated in the candidates’ portfolio, beginning with the Initial Portfolio and Portfolio Interview (end of introductory core) for undergraduate candidates and the Skilled Practitioner Portfolio for graduate candidates (end of methods or School Counseling practicum). The process continues with the Developing Portfolio and Interview (end of methods core for undergraduates), and culminating with the Proficient Portfolio and Interview for undergraduate candidates (end of student teaching) and the Distinguished Portfolio and Interview for graduate candidates (end of program, degree requirement). One of the portfolio requirements is the development of a rationale statement for each artifact.  The artifact is then related to a Conceptual Framework strand and its corresponding INTASC Standards. The portfolio process includes reflections and rationale statements to assure that Teacher Education candidates understand and apply the standards communicated to them through the products selected. In addition, the Teacher Education Handbook is a required text for the first core class (EDUC 100) and Admission to Teacher Education for graduate candidates.

The Unit Assessment System demonstrates a circular process in that the Department delivers instruction that identifies skills, allows opportunities for candidates to practice skills, assesses the acquisition of skills, and provides feedback related to deficiencies or mastery of competencies. This process is multi-leveled since it is initiated within each course, developed through each decision point, continued through all field experiences, and is culminated in student teaching with the demonstration/application of all program expectancies.

Multiple formative assessments are described in each course syllabus. The assessment method is aligned with products to demonstrate competencies appropriate to all levels of the portfolio. Examples of course-linked assessment methods are case studies, lesson plans, presentations, exams, and reflective papers. These methods ensure that the candidate receives maximum opportunities to acquire, apply, demonstrate, and reflect on the necessary knowledge and skills to be an effective teacher.

Formative performance assessment begins with each undergraduate candidate during the second Teacher Education Experience course, EDUC 201 (TEE 2). Each undergraduate candidate’s field placement prior to student teaching is evaluated through the practicum evaluation tool, which is based on the Conceptual Framework as is the graduate candidate’s practicum and/or internship evaluation tool. The Department conducts the first portfolio review/check by using the Initial Portfolio Rubric & Portfolio Interview Evaluation (undergraduate candidates) that allows for assessment of the emerging teacher candidate competencies. An initial formative check of the portfolio is implemented at the end of PSYC 518, Lifestyle and Career Counseling, for School Counseling candidates and methods courses for the Exceptional Needs graduate candidates. At the graduate level the Skilled Practitioner Portfolio Rubric & Portfolio Interview Evaluation is conducted to assess the advanced development of teacher competencies.   The undergraduate Developing Portfolio review occurs near the culmination of the methods courses and prior to the student teaching experience. The candidate is expected to demonstrate growth by completing the Developing Portfolio (part of a continuum of expectancies) as guided by the Portfolio Manual. The methods instructor and a professional education faculty member review the portfolio based on the inclusion of documents with specific criteria that support the acquisition of skills as identified in the Conceptual Framework. Examples include lesson plans, performance artifacts from methods classes with evidence of attention to and respect for exceptional needs students, technology, service learning, and continuing involvement in campus & community outreach (Community Outreach Log), professional development experiences, and field experience evaluation. Faculty members also assess candidates’ summary statements, reflection rationales, and artifacts as evidence of addressing the Conceptual Framework strands and supporting IPSB and INTASC Standards. 
The UAS includes additional assessments related to the Critical Decision Points (Appendix D, Table 2.2a & b). Examples include standardized test scores (Praxis I), field experience evaluations, and overall and professional grade point averages.

The Unit Assessment System is based on the Conceptual Framework and illustrated by The Critical Decision Points for Teacher Education (Appendix D, Table 2.2a & b). It delineates the formative and summative elements according to year of study or program point, criteria /indicator(s), source of evidence, evaluator, and alignment to standards and Conceptual Framework. Each decision point is considered a performance check, and is assessed using the appropriate source(s) of evidence. The student is not permitted to progress further in the program unless he/she has successfully met the criteria for each decision point.

Undergraduate Summative Decision Points Graduate Summative Decision Points

For each summative decision point multiple types of assessment are used. For first summative decision point, Admission into Teacher Education, the candidate is reviewed by multiple evaluators and indicators. The final summative decision point, Completion of Student Teaching, includes the portfolio (Proficient/Distinguished Portfolio Rubric). At this time, the portfolio must be complete with evidence of the proficiencies in all areas of the Conceptual Framework plus the INTASC Principles and NBPTS (advanced candidates beginning 2002). Candidates present their portfolios to a team of professional educators. Professional education faculty, clinical/supervisory faculty, and University supervisors are invited to the presentation for the undergraduate candidates. Graduate candidates invite a school administrator or a mentor teacher, the Program Director, and a member of the Graduate Committee. Artifacts to be included are at the discretion of the student but must show acquisition of knowledge, dispositions, and performances. The structure/format for the Proficient/Distinguished Portfolio is specified in the Portfolio Manual. A rubric is used to assess the candidate’s performance and to ensure consistency and objectivity. In addition to the Proficient or Distinguished Portfolio, the candidate must successfully complete the student teaching or practicum/internship experience. The clinical/supervisory faculty member, school counselor or school administrator completes two evaluation rubrics, one at midterm and one at the end of the placement, plus anecdotal observational reports authored by the University supervisor. Finally, the candidate is expected to satisfactorily complete the specific requirements of student teaching or practicum/internship such as a teaching video, case studies, and samples of (P-12) student learning and assessment.

To provide consistency in the assessment process, the Critical Decision Points (Appendix D, Table 2.2a & b) include portfolio checkpoints (undergraduate) or reviews (graduate) that are evaluated through the use of rubrics. Since the piloting year, rubrics have been revised according to rater reviews (i.e., Professional Education faculty, Clinical/supervisory faculty, University Supervisors, and Advisory Board Members) resulting in refinements related to format and clarity of criteria.

All Teacher Education candidates are required to complete all levels of the comprehensive portfolio process. The Proficient Portfolio is a summative checkpoint within student teaching for undergraduate candidates, as is the Distinguished Portfolio for graduate candidates. Each level builds on the prior level and culminates with the Proficient/Distinguished Portfolio, which is a demonstration of all Teacher Education Program expectations. The portfolio rubric is used to provide consistent feedback to the individual candidate and the program through analysis across all candidates enrolled in Teacher Education. Faculty members in the Teacher Education Programs consider portfolios to be a very effective tool that provides evidence of the acquisition of knowledge, dispositions, and performances necessary for the pre-service teacher and ultimately the professional educator. The portfolio process is designed as an alternative assessment to evaluate the continuous, overall development of the learner. The process becomes a systematic collection of data organized to verify the effectiveness of instruction, and includes artifacts, reflections, self-evaluations, and mentor evaluations of authentic assessment tasks. The following three factors summarize the intent of the portfolio process:

• To enable the candidate to document skills and learning events
• To shift ownership of the learning responsibility to the candidate
• To assess candidate achievement

The Department values the portfolio as a means of assessing learning events and providing evidence of developing skills. The decision points are summative and use the portfolio evaluation process with other methods of assessment to evaluate candidate performance. The cumulative data on candidates provide a reliable judgment and predicts the success of the candidate at each decision point.

Faculty assess both quantitative and qualitative data. Examples of quantitative information recorded include the rubric for portfolio assessment, Praxis Scores, field evaluations, and GPA. Qualitative evidence includes letters of support and the Assessment of Professional Potential from faculty or supervising personnel, writing samples, and practicum and student teaching anecdotal records.

A follow-up study of both student teachers and graduates will be made to determine the reliability of these assessment instruments. This study will provide valuable information on the teacher preparation program itself. The process is a cyclical one in which feedback will lead to program modification to further assure candidate quality. All program changes/modifications are documented in the History of Change document (Table 2.5).

All basic academic and demographic information as well as the quantitative data collected on candidates are recorded in the Unit Assessment System. All decision point requirements are entered in the UAS as met (ü), waived (/), or not attempted or unmet (blank) in accordance with to the data entered for each requirement within each decision point for each candidate. Portfolio evaluations sessions are held each semester. In addition, the Department continues to recruit Arts & Science faculty and new clinical/supervisory faculty to broaden the pool of raters and collect recommendations for more effective evaluation and meaningful feedback on candidates.

Candidates receive individual feedback from their advisors and instructors as they complete each portfolio check or review and decision point. Letters notifying candidates of Admission into the Teacher Education Program and student teaching are sent each semester. Field experiences require a midterm and final evaluation reviewed jointly by the cooperating teacher, the University supervisor, and the candidate. In addition, instructors provide candidates with course feedback (through graded course assignments such as exams, case studies, reflections) throughout the semester. Midterm reports are sent to candidates by the Registrar if their performance in a class is unacceptable (D or F). The Department monitors course performance at the undergraduate level in professional education and issues deficiency points for unacceptable grades. The Department generates letters to those candidates receiving deficiency points that indicate how many points the student has accumulated and the consequences of deficiencies. Deficiency point accumulation is built into the UAS. When a student earns 8 points, a yellow light, and at 11 points a red light, appears in the GPA data point of entry to notify the student and advisor. An accumulation of 12 deficiency points in major/minor and professional education courses may result in non-admittance into or dismissal from the Teacher Education Program. Candidates are only allowed to retake professional education courses once. The following scale illustrates the letter-grade criteria for administering deficiency points.

The Department of Education and the Registrar collaborate to monitor and ensure that candidates meet the decision points. The Department provides the Registrar a database-produced report of candidates’ status. Those candidates not accepted into Teacher Education would not be allowed to register for methods classes because the Registrar would establish a “Department Hold” status preventing registration. At the graduate level, the Admissions Counselor, the Licensing Officer, the Associate Vice President for Adult Learning and University Assessment, and the Department Chair designed a competency-based monitoring system. For example, Admission into Teacher Education must be successfully completed prior to being fully accepted into the graduate Teacher Education program. Again, a “Department Hold” is placed on any graduate candidates who have not been admitted into Teacher Education to block them from registering for graduate courses. Only the professional education faculty can remove a “Department Hold.”

Program audits are conducted a semester prior to program completion by the Director of Student Services for undergraduates and the Associate Vice President for Adult Learning and University Assessment for graduate candidates to ensure all program requirements are met. Another method of monitoring and providing feedback occurs during advising sessions. The secondary candidates meet with their content advisors and the Director of Secondary Education. In these sessions, the Director of Secondary Education reviews the decision points with the candidates. Candidates attending the session initial a sign-in list, which is submitted to the Registrar. Any secondary candidate not listed on the sign-in sheet is put on “Department Hold” status by the Registrar and cannot register for classes. The Department removes the hold status when the student completes the missing indicator(s) for the decision point.

Once the UAS is fully implemented in a secured online database, it will serve three purposes. First, the UAS is a monitoring system for candidates and faculty who will be able to gain access and view program progression. Second, the UAS is an analysis system through which the faculty in the Department will be able to analyze collected data for their advisees’ that relate to program progression and effectiveness through the decision points. Third, the UAS will monitor the progress of teachers and school counselors who are practicing in the field.

The Department determined that some program decisions about candidates and detailed information supporting the UAS would continue to be recorded and stored in individual student hard files. An example of such information is the record of a formal candidate appeal. The documentation of the nature and resolution of the appeal is contained in the candidate’s confidential hard file.

The Critical Decision Points in Teacher Education (Appendix D, Table 2.2 a & b) illustrate the differences between indicators for the initial and advanced candidate levels.

Data Collection, Analysis, and Evaluation

Unit Assessment System Collective Presentation of Candidate Assessments

Since the last NCATE visit, the Department used Microsoft Excel or Access applications to store and organize individual candidate assessment information. Examples of data stored in this early assessment mechanism are the candidates’ status related to the criteria for acceptance into Teacher Education, admission into student teaching, Praxis I and II results, and practicum and student teacher evaluations. The software applications could produce limited reports of aggregated data by specific groups and individual. However, this early system of assessment was restricted in providing the Department with aggregation of overall student performance and program effectiveness. In addition, the Departments databases could not connect to the University databases and use existing information.

In the Fall of 2002, the University supported the development of a customized database for the Department’s electronic assessment mechanism. The move from the Access/Excel-based system to the more comprehensively designed/ programmed model demonstrates the University’s and Department’s commitment to test different information technologies and improve its assessment system. The new electronic UAS program can interact with standard University databases to extract assessment-required information and run individual or group aggregated reports. This system and the reports that can be produced will allow faculty advisors to monitor and analyze student performance and program effectiveness. In addition, the electronic assessment mechanism will avoid the need for the UAS Coordinator and Manager to re-enter or update information extracted from University database. Examples of the data extracted from the University registration module include GPAs, courses attempted, courses completed, grades earned, and student demographic information.

The electronic UAS contains descriptive fields for the purpose of collecting decision point data to monitor individual and aggregated student performance. Targeted criteria obtained through multiple assessments may be entered and stored. The data are used to review and monitor candidate performance so that advisers can inform candidates as to whether or not they are meeting specific benchmarks within specified timelines. With this new technology that sorts and selects student information, the Department can monitor candidates’ progression throughout the program and their induction year(s). If, at any stage of the program, a candidate has not met specific criteria to move to the next stage, he/she is advised of corrective measures. The Department is now capable of analyzing how many candidates met, met with remediation, or failed each decision point through a variety of performance indicators at various levels.  A recent revision to the UAS included an area documenting the remediation recommendation(s), completion date, and result specific to the indicator such as portfolio or Praxis remediation recommendations. Finally, a candidate selection menu entitled “student designations” was developed. The system allows the advisor to select the desired classification options: freshman, sophomore, junior, senior; advisees, licensure, graduate; current and/or former candidates.

The UAS was developed around the Conceptual Framework/Critical Decision Points for Teacher Education (Appendix D, Table 2.2a & b), which is aligned with all standards. All courses within the curriculum with their respective Course Linkage Charts (Appendix E) are entered in the UAS. The purpose of the course charts is to demonstrate the standards-based skills addressed in each course and the assessment method used to provide evidence of skill attainment. Upon successful course completion, the skills attained through each course by the candidate can be documented through a report generated by the UAS (Candidate Performance Alignment Report). The Course Linkage Charts provide one more type of assessment collection in the electronic assessment system to document individual and aggregated performance data among and across candidates. 

The UAS is being implemented in stages. The first stage of the UAS is the student-monitoring phase. This phase includes the entry of the decision point data for undergraduate and graduate candidates. The second stage of the assessment system is referred to as performance analysis phase and is in progress as data continued to be entered. The aggregation of student data is currently being implemented and basic reports are completed at the time of this report. The Department continues to design report requests to be programmed in the system for the analysis of student progress and program effectiveness. The third phase, follow-up analysis, incorporates the external assessments (e.g., graduate follow-up survey) and aggregates data to analyze program effectiveness and beginning teacher performance. Existing surveys are under review and revision. The revised follow-up survey will be disseminated during the 2003 academic year.

Candidate assessment data are presented using multiple assessments from internal and external sources. The following table provides examples and their sources that demonstrate candidate performance related to the Admission to Teacher Education decision point.

The other decision points have a similar variety of sources of internal and external evidence to assess the performance indicators.

The Department incorporates information from two public state reports into the UAS to track candidates’ performance on standardized assessments. First, The Indiana Mentorship Report is an annual state report that indicates whether or not program completers are successful in their beginning teacher induction program. The beginning teacher results are organized into an Access database and will be imported into the UAS. Since beginning teachers have two years to successfully complete their induction, there have been no failed results reported from 1998 –2001 (Table 2.3). Second, the Title II Report is a public report delineating the percent of candidates who successfully completed the required licensing exam, Praxis I. The candidates’ scores are entered into an Access table which collects assessment pass-rate data on a semi-annual basis (Table 1.2). Finally, the Department receives reports from Educational Testing Service each time a candidate takes a Praxis exam (Praxis I or II). Together this information is collected and entered into the UAS by the Department administrative assistant to identify the percent of candidates who successfully complete the state required licensing and induction program.

Indiana Mentorship Report 1998-2001

Table 2.3 

The Refinement and Revision of the Conceptual Framework and Programs
Utilizing the Critical Decision Points for Teacher Education (Appendix D, Table 2.2a & b), the Department established a timeline (Table 2.4) for reviewing data relative to individual candidate’s performance, course and program effectiveness, and overall Department success. Based on these reviews, appropriate refinements will be made to the Conceptual Framework and programs.

Timeline for Data Review

Table 2.4

Assessments from individual candidates are aggregated as follows: 1) the CDP identify the type of data to be aggregated in addition to source of the evidence producing the data; 2) the Coordinator and Manager enter these data in the Unit Assessment System each semester prior to registration.

The electronic system is being programmed to run reports to aggregate and analyze the data that are entered by semester (e.g., candidate status in Teacher Education) or annually (e.g., program completers and recommendation for licensure). For example, candidates must successfully complete the Initial Portfolio and the introductory core courses prior to Admission into Teacher Education. The UAS will provide an analysis of candidate performance which will determine whether the prerequisites are effective predictors of program success. If the correlation between the prerequisites and candidate success in the teacher education program is not positive, the Department will need to consider modification of the perquisites.

Course evaluations provide another source of data. Candidates complete the evaluation and the faculty member receives aggregated results. Faculty members receive these aggregated evaluations each semester and are able to reflect on their own performance to refine and revise their courses.

Next, at the program level a compilation of information gathered through the Teacher Education Program: Candidate Exit Survey is shared with the Department members on an annual basis to consider course and/or program revision (Table 2.5). Program satisfaction surveys are completed at the end of each semester with field-based administrators and clinical/supervisory faculty. The surveys provide perceptions of the effectiveness of the field experiences.  An example of changes resulting from survey input is presented in Table 2.5.  The Department annually reviews external information from the “Title II Report and Indiana Mentorship Report.”

Finally, the University annually conducts the campus wide Student Satisfaction Inventory (Noel Levitz) to assess student’s attitudes regarding academic content, advising, and instructional effectiveness. The collected data are organized by program so each department/program can view the satisfaction level of its majors and compare results with majors across campus. All recommended changes that result from such assessment are sent to the Teacher Education Committee and Teacher Education Advisory Council. Any revisions that have curricular ramifications are forwarded to the Academic Council for approval. The History of Change document (Table 2.5) includes all program changes and when they occurred.

The Department uses multi-leveled, internal, and external candidate assessment instruments and other sources to gather data for the UAS. Examples of these sources were presented above. Aggregated data comparing candidates and programs are compiled from these assessment sources. These data are used to make revisions in Teacher Education programs, which are recorded in the History of Change within the Department of Education document (Table 2.5). The changes stated in the History of Change document occur as a result of the assessments mentioned earlier (e.g., course evaluations, practicum and student teaching evaluations, and surveys). The source, rationale, and date for each change are noted in the document. Recent changes are as follows.

Examples of Change within the Department of Education - 2001 and Beyond

Table 2.5

The Department continues to review and revise the Teacher Education programs based on feedback from a variety of assessments. The effectiveness of its data system will increase as the phases of UAS implementation are completed. As demonstrated above, changes, modifications or innovations have already occurred in the Conceptual Framework and programs. The UAS provides important support for these changes.
 

Management of the Assessment System
The Department of Education continues the process of importing its original Access/Excel-based UAS to the new electronic database system.  The new system provides browser-based access to a comprehensive SQL Server database. The system is designed to allow the Department to extract data from the existing University database to assist with the entry of data into the UAS. There will still be a need to record information not captured in the registration system such as Praxis I & II scores.

However, the ability to use existing internal databases that contain basic information (demographics, GPA, number of hours attempted and completed comprehensive lists of courses and grades, status, and level) will reduce the amount of data entry required of the UAS Coordinator and Manager. The Department of Education is entering information relative to the decision points on all candidates beginning in the Fall of 2001. The student cumulative files are in the process of revision, with the intent of organizing the student files to reflect the decision points. These efforts will allow the collection, analysis, and evaluation of different forms of information/assessment, so as to include both an electronic and a traditional file system.

Three new positions were created to support the UAS. First, the secretary’s role was upgraded to administrative assistant, spurred in part by the need for a UAS data manager and, as such, data management skills were included in the search. Second, the Department hired a UAS Coordinator. This position was approved and filled in the Fall of 2002. The Chair of the Department and the UAS Coordinator will be responsible for overseeing the UAS, with validation and documentation processes the responsibility of all Department faculty. Third, during the academic year 2001 the University continued to support with an additional office assistant. There will be on-going monitoring of the system management. The following chart illustrates the implementation process of data management.

The Administrative Assistant, UAS Coordinator, and the Chair of the Department meet twice monthly with the Database Administrator to discuss, plan, and implement the necessary technology and procedures to provide security and confidentiality for the system. Data are secured through the use of a firewall and SSL 128 bit encryption.

Professional education faculty members have access to the UAS data and the University registration module to assist in monitoring student progression. Faculty members use their network access codes, which are provided once the prerequisite training session to access the assessment system and the registration module. Candidates will be allowed password protected UAS access to monitor their progression in the program. They will have limited viewing access, and no editing privileges.

The Department’s UAS interfaces with the overall institutional assessment plan. The UAS is a successful and comprehensive endeavor to illustrate student performance; it is being considered as a prototype for other academic department’s assessment system.
 

Review and Revision of the Assessment System
Even though the Department is in the early stages of making the UAS operational, there is a “debugging” level of review and revision occurring between the Department and the Campus Computing Center. As evidence, a timeline table was designed to capture UAS problems, problem priority, potential solutions, and date of resolution. The timeline will be maintained as documentation that the UAS is in a continuous state of review and revision. The process for review and revision is based on input from the Teacher Education Committee and the Teacher Education Advisory Council.

The Department continues to refine the decision points resulting in on-going revisions of the UAS. An example of such revision is the development of an editing note that allows the Coordinator to refer the user to an alternative source (candidate file) for the portfolio rubric. The revised system reflects the portfolio requirement with a “met” status and a notation to see the candidate’s file for the rubric. Another refinement of the UAS is the link to an Internet site that searches the State database for diversity demographics.

Use of Data for Program Improvement

The Refinement and Revision of the Framework, Courses, & Curriculum

The UAS uses multi-level, internal and external assessment instruments and other sources to gather data and maintain a systematic and comprehensive assessment plan. These tools and sources provide information on the qualifications and competence of candidates as well as program effectiveness. Examples include:

• Initial, Developing, and Proficient Portfolio checks (initial level candidates)
• Skilled Practitioner and Distinguished Portfolio reviews (advanced level candidates)
• Praxis I and II results
• Course evaluations
• Practicum, student teaching evaluations
• Teacher Education Program: Candidate Exit Survey
• Senior Exit Interview
• Cooperating Teacher/Administrator Evaluation Survey
• Graduate Follow-up Survey
• Beginning Teacher Induction Survey and Teacher Performance Evaluations for graduate candidates working in the teaching field

Aggregated data across candidates and programs are compiled from these assessments and are used to make revisions in the Teacher Education program.  They are then recorded in the History of Change within the Department of Education document (Table 2.5). Curriculum revisions made during the 2002 academic year reflect the Department’s response to standards’ expectancies and candidates’ evaluation of their program. The following examples illustrate the Department’s use of data to improve its programs:

• The Candidate Exit Survey 2001 responses indicated that all candidates desired more behavior and classroom management instructional time. As a result, SPED 430, Classroom/Behavior Management, was added as a requirement to the Elementary and Secondary programs of study.
• In the same survey, working with candidates with exceptionalities was rated “low” in terms of quantity of instructional time by elementary and secondary majors. The new curriculum (Fall 2002) responds to this deficiency with the dual licensure programs.
• Another curriculum change was to increase all student teaching clinical experiences from ten to fifteen weeks.
• It was also noted in the Program Completer Survey that candidates did not fully understand the activities/opportunities for professional development. The Department is reviewing the courses (methods and advanced practicum) in which professional development is a significant component so as to better articulate professional development plans and review of performance.
• Finally, two assessments were recently implemented to evaluate candidate dispositions entitled the Assessment of Professional Potential.

Candidates are provided individual feedback based on the assessments used to make decisions respective to their performance and related to their progression through the program. Candidates are given feedback during each course, at each decision point, and during advising sessions. The new Assessment of Professional Potential evaluations have been implemented in targeted classes. They are reviewed with the candidate and recommendation/remediation is offered at the time of the feedback meeting. In addition, candidates will be able to access their individual file to view their performance in relation to the Critical Decision Points (Appendix D, Table 2.2a & b) through a secured UAS web-based system.

Faculty members receive feedback and reflect on their performance at the course level through course evaluations conducted each semester. The Department Chair completes an annual evaluation of each member and gives feedback during an individual, private meeting. At the program level, a compilation of information gathered annually through the Teacher Education Candidate Exit Survey and Field Experience Effectiveness Survey are shared with the Department members. Finally, the University conducted a campus-wide survey, Student Satisfaction Inventory (Noel Levitz), of student attitudes regarding academics, advising, and instructional effectiveness.  Student responses are collected and analyzed by department, and the results disseminated to all departments.

The systematic feedback provided through assessments, such as those noted, allow for course improvements through course and program evaluations. In addition, the Department continues to review the effectiveness of its data system as it implements the UAS.