Educator Preparation Program (EPP) Accreditation

Measure 1: Completer Effectiveness

As part of our continuous improvement efforts, Mayville State University’s EPP uses multiple instruments to measure completer effectiveness and impact on P-12 learning and development.  Indicators of teaching effectiveness are measured in a variety of ways, including completer surveys, supervisor surveys, and supervisor evaluations.

The EPP’s use of multiple measures provides a more complete description of student growth and completers’ success in the classroom and followed recommendations of best practices to investigate teaching effectiveness.  For further information, please refer to the manuscripts found in the attachments at the end of this measure.

Each spring, findings on completer satisfaction are collected from a survey administered to teachers the academic year following their graduation.  All completers/graduates are invited to complete the Transition to Teaching Survey (TTS), but those who are currently teaching complete an additional section to rate the quality of their preparation.  The TTS asks Mayville State University (MSU) graduates to evaluate their perception of the quality of their instructional practices, their ability to work with diverse learners, ability to establish positive classroom environment, and level of professionalism.  The survey is administered approximately one year after the graduates complete their preparation program.

On the survey, completers’ satisfaction with their preparation being relevant to the responsibilities they confront on the job and their preparation being effective are indicated with tends to agree (3) and agree (4) ratings.  Mayville State University’s Teacher Education Program has set an acceptable target of an average of 3.0 or higher on all indicators (all tends to agree or agree).  EPP results are also compared to North Dakota and super aggregate results to ensure commensurate scores, ultimately aiming for higher scores than both the North Dakota and super aggregate.  Results of the survey are reviewed annually by the Division of Education and Teacher Education Committee faculty.  The most recent results can be found in the table below:

In the spring of 2022, the EPP reached out to all graduates from the previous academic year and asked them to complete the common metrics Transition to Teaching Survey (TTS).  The EPP chose to add several supplemental effectiveness questions to the end of the TTS to gage our completers’ impact on P-12 learning and development.  One of the last questions on the TTS asks participants if they would be interested in sending their supervisor evaluations from their most recent academic year.  Of the 55 respondents to the supplemental effectiveness questions, 26 indicated they were willing to submit supervisor evaluations.  When the follow-up request was sent, 12 candidates submitted.  Of those 12, all were undergraduate students (BSEd) and no graduate students (MAT) submitted evaluations.  The 12 undergraduate students included 2 early childhood graduates, 6 elementary education graduates, 3 secondary education graduates, and 1 special education graduate.  Interestingly, two of the graduates were teaching in a content area outside of what they were prepared to teach.  One early childhood and one secondary education graduate were teaching in Special Education classrooms.  No MAT graduates sent supervisor evaluations to the EPP for analysis.  All 12 completers who submitted supervisor evaluations received favorable evaluations from their supervisors. A qualitative analysis of results indicated the following:

• Overall Early Childhood Education Findings (2022): Only one Early Childhood Education major, teaching in an Early Childhood setting, participated in the study in Spring of 2022 and therefore, generalizations about the data cannot be made.  The participant’s supervisor felt the Mayville State graduate was effective in aligning learning targets with standards, using resources to design coherent instruction while planning for assessment and differentiation, communicating learning targets effectively, and facilitating activities that promote high cognitive engagement.  The Mayville State graduate also received "effective" ratings in creating a respectful classroom culture and establishing clear expectations for behavior.  As related to professionalism, the graduate received "effective" ratings in reflecting on their teaching practice, engaging in professional development, maintaining professional responsibilities, and communicating with families.  These "effective" rating categories are consistent across Elementary Education majors as well who take the same professional education and methods courses.
  
  
• Overall Elementary Education Findings (2022): Six Elementary Education majors submitted their supervisor evaluations, reflective of the 2021-2022 academic year. All six graduates are teaching in an elementary classroom: 3 in Kindergarten, 1 in a second/third grade combo, 1 in third grade, 1 in fourth grade, and 1 in fifth grade. Most supervisor evaluations assessed overall performance while two were from a math lesson, and one came from a writing lesson. Consistent themes across participants arose as data was coded and analyzed. These common themes were formed when three or more supervisors agreed that graduates' performance was proficient or higher in any particular category. Most ratings within the supervisor evaluations were related to effective planning and preparation for student learning. Supervisors felt that Mayville State graduates were effective in developing rigorous learning goals, aligning learning goals to standards, using resources to design coherent instruction while planning for assessment and differentiation, communicating learning targets effectively, facilitating activities that promote active, cognitive engagement, and designing varied assessments to advance student learning. The Mayville State graduate also received "effective" or "proficient" ratings in creating a respectful classroom culture, establishing clear expectations for behavior, managing classroom procedures, and implementing systematic ways to establish and modify classroom rules and procedures. As related to professionalism, the graduates received "effective" or "proficient" ratings in reflecting on their teaching practice, engaging in professional development (both formally and informally), maintaining professional responsibilities, and communicating with families through a variety of outreach activities. An inconsistency between ratings appeared in two graduates' ability to chunk content into digestible pieces. One candidate was effective and the other was developing skills in this area. Another area that was only rated by one supervisor was graduates' ability to teach students to self-regulate their behavior. This is a challenging skill and develops with time, but interesting none-the-less.  When looking for patterns in the data, there was very few supervisors who included ratings related to graduates' ability to build relationships with students in the Spring 2022 results.  Ratings related to establishing rules and procedures were evident again this year in the supervisors' evaluations.  No comments were made this academic year related to COVID-19 or the disruption the pandemic causes in graduates' ability to meet the demands of teaching.
  
  
• Overall Secondary Education Findings (2022): Three secondary education majors consented to participate in the study, however, only two of them were currently teaching in high school classrooms, one math, one social studies.  Because of the low number of participants, generalizations about the data cannot be made.  However, there were items that were consistent across both graduates' supervisor evaluations.  The supervisors felt the Mayville State graduates were effective in providing and communicating clear learning goals while planning and preparing for learning.  Supervisors also agreed the graduates used structured practice sessions to deliver instruction.  This was a common theme from last years' data analysis as supervisors agreed graduates were effective at delivering instruction.
  
  
• Overall Special Education Findings (2022): One true Special Education major (who also double majored in Elementary Education) submitted their supervisor evaluations, reflective of the 2021-2022 academic year.  Two other completers, one an Early Childhood major and one a secondary Math major, are currently working in a Special Education classroom, but were not initially prepared by Mayville State University to teach Special Education.  Two different evaluation frameworks were used.  The Special Education teacher, trained in Special Education, was evaluated using a school-created performance assessment with the levels of Awareness, Knowledge, Skill Practice, and Skill Application with the latter being the highest rating.  The other two evaluations were based on the Danielson framework and used the headings of Unsatisfactory, Basic, Proficient, and Distinguished with the latter being the highest rating.  Consistent themes across participants arose as data was coded and analyzed.  These common themes were formed when all three supervisors agreed that graduates' performance was proficient/skill practice or higher in any particular category.  Supervisors felt that Mayville State graduates, working in a Special Education classroom, were effective in knowing students' special needs, establishing routines and procedures to help students be successful, and being aware of students' behaviors and responding appropriately to misbehaviors.  The Mayville State graduate also received "proficient" or "skill practice" ratings in effectively engaging learners and using appropriate instructional methods. Inconsistencies between ratings appeared in several indicators.  In two of three evaluations, supervisors agreed that Mayville State graduates were competent in designing learning outcomes for diverse learners, collaborating with families, and making use of available technology.  In all three evaluations, supervisors rated graduates' ability to use student achievement data to drive instruction and developing appropriate instructional objectives.  In both descriptors, one supervisor rated one candidate as "basic". A final finding arose as the following criteria, which are all important indicators of an effective Special Education Teacher, only appeared only once across all supervisor evaluations: using available resources, collaborating with team members and stakeholders, and engaging in professional growth opportunities.  This might have been due to different evaluation frameworks, but it is worth exploring, none-the-less.

One final, comprehensive measure related to completer effectiveness is the case study research that the EPP has engaged in.  As a strategy to assess the effectiveness of completers in the classroom and impact on P-12 student learning, case study was used as an alternative to value-added measures (unavailable at the state level in North Dakota) to holistically examine complex attributes of quality teaching (Merriam, 1998; Yin, 2014).  The Reflective Experiential Teacher framework, developed by the EPP, was used to design methods and guide analysis.  The constructivist framework incorporates the knowledge, skills, and dispositions of the ten core teaching standards of the Interstate Teacher Assessment and Support Consortium (InTASC) (CCSSO, 2013) divided into the four CAEP Standard 1 categories of: Learner and Learning, Content, Instructional Practice, and Professional Responsibility. 

This case study was part of a larger investigation of the role EPPs play in graduates’ teaching effectiveness, among a myriad of other complex factors, and the role graduates’ effectiveness has on student learning.  Data was collected to answer the questions:

• Do P-12 students in the classrooms of EPP completers demonstrate expected levels of learning and development?
  
  
• What factors do graduates perceive impact students’ learning and development? and
  
  
• How can a EPP improve program quality based on results of P-12 student learning in classrooms in which graduates are teaching?

Mixed methods data collection in the descriptive study included: graduate and supervisor surveys, self and supervisor evaluation of skills and dispositions, interviews, document review, student engagement surveys, achievement data, and pre and post assessment data. Descriptive statistics were used to describe data for each case.  Within each case, the constant comparative method of qualitative data analysis was used (Glaser & Strauss, 1967) to construct codes, categories, subcategories or themes through continuous comparison of data (Merriam, 1998, p. 159).  Validity and reliability of measures within the case study are described in instrumentation in the manuscripts below.

Results indicated most students from participants’ classrooms demonstrated learning growth, and participants viewed their teaching as effective.  Implications for programs include pre-service opportunities to identify and respond to authentic student engagement, collaboration with colleagues, the impact of reflection on teaching practices, and opportunities to build collective teacher efficacy.  Additional insights describe pre-service teacher training to measure student progress and measure impact based on classroom assessments. The investigation provides a replicable case study design for teacher educators to examine relationships between teacher preparation, program graduates, and P-12 student outcomes.  Findings and results are presented in the finished manuscripts available on this webpage as attached files; these include detailed accounts of recruitment, methodology, research questions, and analysis.

Data cycles started with protocol development in 2016 followed by purposeful sampling of participants, implementation, refinement, and further data collection. A pilot study in Spring 2017 included two elementary completers. In Spring 2018, two completers from elementary education and two from secondary science education participated.  To represent all EPP program areas, participants in Spring 2019 were recruited from the remaining areas of early childhood, special education (SPED), and MAT. Recruitment scripts were sent to 11 eligible completers (early childhood n = 6; SPED n = 3; MAT n = 2); none agreed to participate. Of the three SPED graduates, only one was currently teaching in a special education position. All others were employed in a regular education setting. The one SPED completer teaching in a Special Education classroom did not consent to participate. The completers in MAT and EC noted their reasons for not participating: limited time due to other school-related obligations (i.e., coaching, assessment), personal responsibilities, and need for supervisor participation/consent. Further complications for participant recruitment came in March 2020 when the COVID19 pandemic shut down schools across the state and nation.  Because of the stress teachers were under and challenges they were experiencing (and continue to experience), the EPP decided to suspend the case study research and collect data that would regularly be collected from graduates including the Transition to Teaching Survey (TTS) and Supervisor Survey (SS). The TTS includes a question inquiring whether first year teachers would be interested in participating in further data collection to measure their impact on P-12 learning and development.  Of the 44 participants who completed the TTS, 15 first year teachers agreed they would be willing to send the EPP data manager their two most recent supervisor evaluations.  Again, due to the COVID19 pandemic in spring 2020, the EPP decided to postpone data collection from participants.   Beginning in the Spring of 2021, the EPP will continue previous efforts to collect annual measures examining completer performance in different grade levels and content areas.

Also of note, the EPP typically receives a “First Year Teacher Effect on Student Outcomes” report from State Longitudinal Data System (SLDS) in early spring.  This report shares information related to SLDS data for demonstrated growth in math, reading, and language arts of students in Mayville State’s completers’ classrooms.  The data is compared to North Dakota non-first year teachers and other North Dakota first year teachers and is shared visually.  Due to the COVID19 pandemic and closing of schools, SLDS did not collect comparative data for the 19-20 school year and therefore, the report was unavailable this spring and last spring.  Either way, the EPP continues to find the state-provided data as inadequate for evaluating program impact due to a limited sample, the inability of state data to be disaggregated by program or directly linked to completers, and test scores limited to reading, language arts and math which accounts for only a portion of EPP completers. In addition, only completers who remained in ND to teach have student data available.

Overall, the EPP continues to determine that state provided, teacher-linked student achievement data does little to inform continuous improvement efforts, and thus the EPP continues to pursue other sources of evidence (i.e., teaching effectiveness survey questions, supervisor evaluations, and case study research).

In conclusion, multiple measures support the fact that Mayville State University Educator Preparation Program completers' students did indeed demonstrate expected levels of learning. As a descriptive case study, the EPP did not attempt to infer direct causality of teacher preparation or effectiveness of teaching on the results of student outcomes; however, it appears completers contributed to positive student outcomes amid multiple influencing factors. Foremost, results from learner outcomes reaffirmed the EPP’s commitment to ensure every learner has a competent, effective, engaging, and reflective educator.

File Attachments:

1. Anderson, S.K., Hagen, B., Whitsel, C., Dulski-Bucholz, A., Smith, K., & Willeson, A. (2019). Leveraging case study research: A mechanism for measuring teaching effectiveness. Mid-Western Educational
    
2. Anderson, S.K., Hagen, B., Whitsel. C., Smith, K.D., & Duffield, S. (2020). Graduate impact on student learning: A descriptive case study.
    
3. Madler, A., Anderson, S. K., & Smith, K. (2020). Perceptions of teacher preparation for classroom diversity: A secondary data analysis.
    

Measure 2: Satisfaction of Employers and Stakeholder Involvement

Each spring, findings on employer satisfaction are collected from a survey administered to supervisors of first year teachers who graduated from Mayville State University (MSU).  MSU graduates complete a Transition to Teacher Survey (TTS) where contact information for their supervisor is collected and used to administer the Supervisor Survey (SS).  The SS asks supervisors to assess the quality of graduates’ instructional practices, abilities to work with diverse learners, abilities to establish positive classroom environment, and levels of professionalism. The SS is administered to direct supervisors, mainly principals, of EPP graduates employed in schools approximately one year after the completion of their preparation program at Mayville State.

On the survey, employers’ satisfaction with completers’ preparation for their assigned responsibilities in working with P-12 students is indicated with tends to agree (3) and agree (4) ratings.  Mayville State University’s Teacher Education Program has set an acceptable target of an average of 3.0 or higher on all indicators (all tends to agree or agree).  Results of the survey are reviewed annually by the Division of Education and Teacher Education Committee faculty.  The most recent results can be found in the table below:

Stakeholder involvement is an integral part of the success of Mayville State Teacher Education graduates.  With over 340 MOUs with schools in the region, state, nation, and even international connections, the EPP involves stakeholders in program design, evaluation, and continuous improvement processes.  This is done in several ways including displaying CAEP accountability measures on our website, meeting bi-annually with our Teacher Education Advisory Committee, and consistent communication with administrators and teachers when designing mutually beneficial partnerships and field experience placements.  Over the last academic year, the Teacher Education Advisory Council met and discussed feedback on a variety of topics.  Evidence to support stakeholder involvement, including TEAC agenda, presentation, and summary (minutes available upon request), can be found below. 

TEAC December 2021-Agenda

TEAC April 2022-Agenda

Communication and collaboration with partners occur in both structured and organic ways.  Mayville State University’s Teacher Preparation Program implements a variety of field experiences where our teacher candidates can apply theory to practice.  We nurture our partnerships and have consistent relationships with our partners each year while allowing room to develop new partnerships that are mutually beneficial to the partner and our teacher candidates.  Below, you will find a list of several partnerships implemented over the 2021-2022 academic year, a brief description of each, and how stakeholders were involved in the design and implementation:

• Block 1 Field Experience – Early Childhood and Elementary Education majors participate in this field experience as part of their first block of methods classes.  Third graders from the local elementary school in Mayville, ND are invited to campus where mini-classrooms are created to welcome them for 2.5 weeks of learning.  Teacher candidates apply methods of instruction for reading, math, and science aligned to state standards selected by clinical partners, the principal and third grade teachers.  The faculty members who teach in these methods classes work closely with the school administrators and teachers to design a field experience that is mutually beneficial for both the school district and Mayville State.
• Block 2 Field Experience – Early Childhood and Elementary Education majors participate in a 5 ½ field experience in our local elementary schools. Each Block 2 student is paired with a cooperating teacher that they work from 8:00-11:30. On Mondays, Tuesdays, Wednesdays, and Thursdays, our students work with their cooperating teachers to pull small groups, design whole-group lessons, provide interventions, etc. Since our Block 2 students have their North Dakota Substitute Teaching License, they are provided the opportunity to sub in their field experience classrooms. This gives our students opportunities to design and implement sub plans. This also gives the classroom teachers opportunities to work in their PLCs, collaborate with other teachers, and plan future lessons. On Fridays, the students return to Mayville State's campus to debrief their experience.
• General Methods Field Experience – General Method students are paired with a cooperating teacher in an area school to complete a 30-hour field experience. During this field experience, they are to implement strategies they are learning during their General Methods course. They are required to write lessons plans and teach two lessons. They record these lessons and debrief with their General Methods instructor and the Field Mentors also provide direction on what lessons to teach and feedback on how the lessons went.
• Secondary Methods Field Experience – Secondary Methods students participated in an immersive week long observation/field experience learning opportunity at various schools across the state and region. Students were actively involved in student and faculty life during their time in their school placement. As a result of full day observations, student feedback was very positive. The teacher candidates had time to build relationships with both students and teachers. The experience of spending five consecutive days allowed our candidates to be immersed into the world of full-time teaching.
• Learning Disabilities Practicum – Students seeking a Special Education major or Special Education minor participate in this field experience as part of their requirement for their program of study. Students are required to complete 60 hours of field experience in a special education classroom serving students with learning disabilities. Teacher candidates apply methods of instruction provided in their Learning Disabilities course. The faculty members that teach the course work closely with the school administrators and teachers to design field experiences that are mutually beneficial for both the school district and Mayville State teacher candidates.

Measure 3: Candidate Competency at Completion

The table below outlines the number of candidates admitted to the program and enrolled at Mayville State during the 21-22 academic year in addition to the subset of completers. 

The table below identifies academic measures that provide data on candidate competency.  These measures include Praxis exam data and GPA.

The table below identifies non-academic measures that provide data on candidate competency.  These measures include Skills of Teaching Observation Tool (STOT) and Disposition Evaluation results from the student teacher cohorts during the 2021-2022 academic year.

The link below provides evidence of Mayville State University Teacher Education Program’s submission of the annual Title II report:

Title II Report-2023

Measure 4: Ability of Completers to be Hired

In the spring of 2022, the EPP reached out to all graduates from the previous academic year and asked them to complete the common metrics Transition to Teaching Survey (TTS). In addition to addressing completer impact, the TTS asked licensure and job status questions. The survey indicated that 98% of completers had applied for a professional teaching license. Of those who had applied for a professional license, 89% applied for a North Dakota license, and 16% applied for a Minnesota license. When asked in which state(s) do you hold a teaching license, 87% indicated North Dakota and 15% indicated Minnesota. Three completers (5%) applied for a job outside of teaching and stated, "a limited number of teaching positions available in my field", "more future prospects outside of teaching" as the most common reasons.

When asked if they sought employment as a licensed teacher, 96% of completers indicated "yes". Most students (39%) indicated they had one job interview while 29% of students had 2-3 job interviews. When asked how well prepared they think they were for their teaching job interviews 85% of students felt prepared. For those that completed an interview, 90% reported that they were offered teaching positions and accepted them.

The following table depicts EPP completers’ current employment situation: