2a.1. How does the unit ensure that the assessment system collects information on candidate proficiencies outlined in the conceptual framework, state standards, and professional standards?
The unit ensures collection of information on candidate proficiencies outlined in the conceptual framework through assessment of signature assignments. Signature assignments are key assessments of course-based proficiencies demonstrated by candidates’ in professional education courses and program-specific courses. Signature assignments also include key assessments of candidates’ proficiencies. These proficiencies are demonstrated in the series of field-based assessment activities known as the Candidate Field Assessment Process (CFAP). All proficiencies assessed through both course-based and field-based signature assignments are aligned with the unit’s conceptual framework. Proficiencies include knowledge, skills, and dispositions expected of candidates in baccalaureate and post-baccalaureate, initial and advanced licensure programs.
Detailed tables of signature assignments’ alignment with the conceptual framework are found in Conceptual Framework Alignment Tables, 2003-2008 (Exceptional Needs, School Counseling, baccalaureate programs), Conceptual Framework Alignment Tables, 2009 (Exceptional Needs, School Counseling, baccalaureate programs) and Conceptual Framework Alignment Tables, 2010 (Exceptional Needs). In the current institutional report, alignment tables are based on the 2003-2008 conceptual framework for the unit’s data analysis.
The unit ensures comprehensive collection of information on candidate proficiencies outlined in state standards through alignment of professional education courses and program-specific courses with Indiana Department of Education’s (IDPE) state standards. Knowledge and performance proficiencies emphasized in courses are detailed in course matrix charts (baccalaureate, exceptional needs, school counseling). Charts for all program courses will be included in the finalized electronic exhibit room. Additionally, course matrix charts designate the evidence used by instructors to assess candidates’ acquisition of proficiencies. The totality of courses that comprise the unit’s programs are configured in curriculum alignment matrices (baccalaureate, exceptional needs, school counseling). Matrices for all programs will be included in the finalized electronic exhibit room. These matrices illustrate the frequency with which state standards are addressed in courses. Also, curriculum alignment matrices indicate the alignment of the unit’s conceptual framework with state standards. Finally, curriculum alignment matrices indicate how state standards are aligned with professional standards. An example of proficiency from the curriculum alignment matrix of school counseling program, School Counseling Professionals, Standard 1 is: The school counselor has the knowledge of change theory and educational reform. Also noted on the matrix is Standard 1’s alignment with the unit’s conceptual framework, and standards from the American School Counselor Association (ASCA).
The unit ensures a comprehensive collection of information on candidate proficiencies outlined in professional standards. At the baccalaureate initial licensure the dual program level and post-baccalaureate initial licensure level, candidate proficiencies are aligned with the INTASC
Standards and the Council for Exceptional Children (CEC). At the post-baccalaureate exceptional needs advanced licensure level, candidate proficiencies are aligned with INTASC, NBPTS, and CEC. School counseling standards are aligned with standards from the American School Counselor Association (ASCA).
2a.2. What are the key assessments used by the unit and its programs to monitor and make decisions about candidate performance at transition points such as those listed in Table 6?
2a.3. How is the unit assessment system evaluated? Who is involved and how?
The Unit Assessment System (UAS) is evaluated by professionals external to the unit through the 2008 State of Indiana UAS Review Process. Representatives from higher education institutions in conjunction with the Indiana Department of Education evaluated the capacity of the unit’s assessment system to generate data useful for making improvements in candidate performance, program quality, and unit operations.
The unit’s external advisory councils are involved in evaluating the UAS. Advisory councils, such as the Teacher Education Advisory Council (TEAC) and the School Counseling Advisory Council meet twice annually to review program decisions, such as revision of the conceptual framework.
Unit personnel and the University Technology Services (UTS) evaluate the capacity of the UAS to collect and process useful data from multiple assessments. Unit and UTS personnel monitor the data collection and processing to ensure it is efficiently and successfully integrating university-wide systems, such as Jenzabar, with unit-specific technology, such as the Candidate Tracking System (CTS) and TaskStream. An example of such collaboration between the unit and UTS occurred as they developed the in-house CTS. The CTS interfaces with the university’s student management system, Jenzabar, to enable the unit to track candidate progression through program transition points. The development of the UAS and subsequent modifications are documented in the UAS History of Change.
The unit employs a full-time UAS manager and UAS coordinator to oversee the functioning of unit assessment system. Together with the department of education chair, they are responsible for ensuring that information is collected and processed for analysis in a timely, efficient manner. They evaluate the capacity and efficiency of the assessment system to ready information the unit uses to make decisions on candidate performance, program quality, and unit operations.
Unit faculty evaluates the utility of UAS information on candidate performance. The unit’s early Excel-based data processing and reporting mechanisms were deemed less than optimal. The data entry features lacked the flexibility required to generate meaningful reports. In response to faculty’s unfavorable evaluation of the Excel-based system, the unit adopted TaskStream to track candidate performance on course and field-based signature assignments.
Also, the unit collaborates with the university’s Office of Institutional Research and Effectiveness (OIRE) in evaluating the UAS. A prominent illustration of collaboration among OIRE, UTS, and unit occurred in 2008-2009 with investigations of data management systems that integrate university-wide and unit-specific technologies. The university required a data management system to assess program quality. At the same time, the unit required a data system other than TaskStream that would interface with the university’s student management system, Jenzabar. Collaborative investigation among UTS, OIRE, and the unit led to the adoption of Blackboard Outcomes. Changes resulting from evaluation of the unit assessment system are detailed in the UAS History of Change document.
2a.4. How does the unit ensure that its assessment procedures are fair, accurate, consistent, and free of bias?
The unit ensures fair procedures by providing detailed, clear directions on signature assignments directions. Rubrics and field evaluation instruments are reviewed in courses and posted for electronic access, affording candidate ready access early in the assessment cycle. The unit consults advisory council partners to develop and refine instruments. The unit solicits input on field assessment instruments from field practitioners, so as to better understand the types and levels of performances that can be fairly expected of candidates. Fairness is also ensured through conditions under which assessment occurs. Although traditional paper-pencil tests are administered, many assessments are project-based. As such, candidates are not subjected to conditions, such as noise, lighting, and improper facilities that might impair their performance.
The unit ensures that its procedures are accurate by aligning signature assignments with conceptual framework performance outcomes. Programs continually review signature assignments and dispositional assessments to ensure that they accurately correlate to expectations set forth in the conceptual framework. To promote assessment accuracy, education faculty collaborated with a nationally-recognized author in dispositional assessment to revise the dispositional instrument. Also, to ensure accuracy in scoring dispositional instruments, field-based and secondary education university faculty were invited to participate in training workshops.
The unit ensures consistent procedures by utilizing a common grading scale on course-based signature assignments in all unit programs. Also, candidate portfolios are co-assessed by unit faculty, while portfolio presentations are co-assessed by faculty from across the university and professional education partners from the community. The multiple-assessor approach leads to consensus results. The school counseling program director monitors scoring of candidates’ disposition performance through meetings with field supervisors.
The unit ensures freedom from bias by soliciting input on assessments from advisory councils and field-based faculty to ensure bias-free language. For example, written assignments and assessment instruments are gender-neutral. Also, unit faculty utilize and encourage candidates to utilize approved terminology when referring to populations that may be subject to biased treatment.
2a.5. What assessments and evaluation measures are used to manage and improve the operations and programs of the unit?
Several measures the unit uses to manage and improve operations and programs are the following: Praxis I, the Noel-Levitz Student Satisfaction Inventory (SSI), Undergraduate Exit Survey, and Individual Development and Educational Assessment (IDEA) course evaluations, and the Field-Based Experience Assessment Survey.
Results from Praxis I were used to change the program point when candidates are required to take Praxis I tests. As a number of candidates did not pass Praxis I, the unit moved the Praxis I requirement to an earlier point in the program, allowing candidates more opportunity to remediate.
Candidate feedback on Noel-Levitz SSI indicated that secondary candidates were less satisfied than elementary candidates with academic advising. These results were used to modify the advising process whereby secondary education content faculty and education faculty share advising duties. Consequently, secondary candidates’ satisfaction with academic advising improved.
Candidates provide data on program quality through results on the Undergraduate Exit Survey. Results from this survey prompted the baccalaureate initial licensure dual programs to add topics such as classroom and behavior management and teaching high ability students to student teaching seminars.
Individual Development and Educational Assessment (IDEA) course evaluations provide feedback from candidates on faculty performance. IDEA results are used to improve faculty performance. For example, faculty formulate annual performance goals based on IDEA results.
Cooperating teachers provide data on program quality via the Field-Based Experience Assessment Survey. On this survey, cooperating teachers evaluate two areas: student teachers’ preparation; and, the level of support the unit provides to cooperating teachers. Feedback received from cooperating teachers indicated their desire for additional guidance in fulfilling their roles and responsibilities. As a result of, the unit augmented its Cooperating Teacher Workshops with vodcasts posted on iTunes University.
Program improvements are informed by feedback from three sources: 1) the Indiana State Department of Education’s (IDOE) higher education Program Review process. 2) the unit’s three advisory councils, which assess the validity and utility of program improvements. For instance, 2008-2009, the Teacher Education Advisory Council collaborated with the unit in revising the conceptual framework. 3) the Teacher Education Committee (TEC). TEC recommends secondary education program changes, such as addition and deletion of courses prompted by adoption of the university’s General Education curriculum.