2b.1. What are the processes and timelines used by the unit to collect, compile, aggregate, summarize, and analyze data on candidate performance, unit operations, and program quality?
The unit’s processes and timelines for collecting, compiling, aggregating, summarizing, and analyzing data for assessment of candidates, programs, and the unit are documented in the Comprehensive Assessments Tracking (CAT) charts (Candidate, Program, Unit). CAT charts indicate sources of data, description of data, the technology and chronology involved in managing the data, the individuals involved in data collection, how the data are summarized and formatted, personnel who supervise the data process, and to whom and when the findings are disseminated. The purposes of assessments dictate the frequency regarding which data are collected and analyzed. Faculty review of candidate performance, unit operations, and program quality occurs annually. Schematics outline candidate and program assessments for which are aggregated and analyzed on annual basis.
Data collected from candidate performances come from course-based and field-based signature assignments. Field-based signature assignments are part of the Candidate Field Assessment Process (CFAP), which is the process by which the Department of Education collects data on candidates’ field performance. Additional data on candidate performance are collected from dispositional assessments and portfolios. The Exceptional Needs and School Counseling Programs use data collected on candidates from admissions interviews to make decisions regarding candidate admission eligibility.
In addition, the unit collects performance data from Praxis I and Praxis II exam scores. At the initial licensure baccalaureate level, Praxis I data are collected to determine candidate eligibility for program admission. Praxis II data determine candidates’ eligibility for state licensing. Several data items are collected on post-baccalaureate exceptional needs and school counseling applicants to determine their eligibility for program admission: grade point averages, GRE/MAT scores, and personal references.
On-going assessments include faculty review of candidate performance on course-based and field-based signature assignments, measures of candidates’ impact on student learning, and candidates’ dispositional performance. Data on candidate performance are analyzed by faculty throughout the semester in their respective courses. Data on candidate performance are analyzed at transition points to determine candidate eligibility for program progression.
The UAS manager collects and summarizes data, while the chair and faculty analyze data. In preparation for the annual Data Decision Day (D3), the UAS manage provides data summaries for faculty review and analysis. During D3, the chair and faculty analyze and interpret data in both written and oral form. Subsequently, faculty use data to inform decisions regarding curriculum, assessment, and instruction.
Data from course-based and field-based signature assignments and from dispositional assessments are summarized and analyzed in report form. Aggregated Praxis I and Praxis II scores are summarized in table form and analyzed by faculty and the unit assessment coordinator during routine department meetings. At the post-baccalaureate initial and advanced licensure levels, the exceptional needs and school counseling admission committees use scoring rubrics to analyze letters of reference, results of applicant interviews, GRE / MAT scores, and undergraduate transcripts.
TaskStream is among the unit’s principal information technologies. TaskStream is used by faculty to score signature assignments. TaskStream data reports are run by the Unit Assessment Manager and used by faculty to summarize and analyze individual candidate performance. TaskStream is also used to collect, aggregate, and analyze candidate performance data by program. TaskStream is augmented by the Candidate Tracking System (CTS) and student files. The CTS is used to maintain individual Praxis and GRE/MAT scores, record candidates’ field placement venues, and indicate candidate program progression. Jenzabar, the university-wide student management system, maintains candidate data that are downloaded to the unit’s CTS.
The unit uses a variety of processes to collect and analyze data on unit operations. The Report Bank, maintained by the university’s Office of Institutional Research and Effectiveness (OIRE), provides data on course enrollment, historical enrollment by major, student retention analysis, and term enrollment demographics by department and major. Noel-Levitz SSI is used to collect data on student satisfaction with unit’s academic advising model. Data from the Noel-Levitz are summarized and analyzed bi-annually. Also, a component of assessment of unit operations is data on staff, faculty, chairs, and dean performance. Performance assessment data is collected annually. On an annual basis the unit reviews technology needs related to hardware and software. The unit’s financial operations are analyzed through monthly and annual budget reports.
The unit uses Microsoft OneNote to document a variety of information used to maintain the unit’s assessment system, such as: meeting minutes, concerns and accomplishments of candidates voiced by faculty, candidate requests or appeals (initial baccalaureate /post-baccalaureate), changes in policies and procedures (History of Change), and proceedings and findings from training workshops. The Candidate Tracking System (CTS) is used by the unit in conjunction with Jenzabar, Report Bank, and BlackBoard to maintain information on field placement demographics, program enrollment trends, and scores on Praxis, GRE, and MAT exams.
Processes used to determine program quality are based primarily aggregated candidate performance data. Data are aggregated from course-based and field-based signature assignments. Additional data on program quality are collected from aggregated scores on candidate dispositional assessments and portfolios. More data are collected from Alumni Surveys, Employer Surveys, Undergraduate Exit Surveys evaluations, field-based experience assessments, Career Services Survey, and IDEA course evaluations.
2b.2. How does the unit disaggregate candidate assessment data for candidates on the main campus, at off-campus sites, in distance learning programs, and in alternate route programs?
The unit uses TaskStream to disaggregate data on candidate performance by program, course, and program transition points. Data from Alumni, Employer, and Undergraduate Exit Surveys are disaggregated by program under the management of the university’s Office of Institutional Research and Effectiveness (OIRE) utilizing SPSS analytical software. The UAS coordinator collaborates with the OIRE to disaggregate data from all accredited programs. Due to the low number of secondary education graduates and low survey response rates, survey results from all secondary education program completers are aggregated. At the advanced licensure level, data are disaggregated by school counseling and exceptional needs programs. Data analysis has prompted revision of the following assessment instruments: formal lesson evaluation, student teaching evaluation, and assessment of professional dispositions.
It should be noted that the unit has only one alternate route program, Transition to Teaching, which is in Secondary Education Art. Currently, the unit has no distance-learning program or off-campus sites.
2b.3. How does the unit maintain records of formal candidate complaints and their resolutions?
The university defines policies for student complaints in the Student Handbook, which is available on the university internet site at http://www.sf.edu/sf/campus and in the academic catalogs. Based on our Franciscan values, which promote respect for all individuals and peaceful resolution of conflicts, the university policies encourages an informal, problem-solving approach. If resolution is not achieved through informal processes, the university has clearly defined policies for formal complaints, which begin when a student submits a written complaint to the appropriate leader. The university has separate policies and procedures for student complaints related to academic issues, student life, and financial aid.
All candidates are encouraged to resolve requests and complaints related to the unit through informal and direct contact with the appropriate faculty or staff member. The exception occurs in cases of harassment and discrimination. In the event of harassment or discrimination, baccalaureate candidates should follow procedures outlined in the University Student Handbook, while baccalaureate candidates should refer to the policy located in the Graduate Catalog).
For cases outside the realms of harassment and discrimination, the unit and Department of Education have established policies for candidates to follow before referring requests and complaints to the university’s administrative level in the form of appeals. Documents related to complaints and appeals are maintained in secure, confidential files located in the offices of the Chair of Education and Chair of Psychology and Counseling, respectively. Access to these files is limited to the unit’s department chairs and Dean of the School of Professional Studies.
Appeals may be made regarding unresolved matters that relate to the application of existing policies, such as course grades. Appeals do not require changes to established policy. There is, however, a university mechanism for requesting changes to policies. Details regarding requests, complaints, and appeals are found in the University Student Handbook, Undergraduate (Baccalaureate) Teacher Education Handbook, the Graduate (Post-Baccalaureate) Handbook.