Proven Course Redesign eAcademy Evaluation

1. Overview

sjsuDuring July and August 2013, the California State University Chancellor’s Office and campus partners offered a series of 3-day Professional Development “eAcademies” for faculty to share existing practices that have successfully reduced repeatable grades in selected bottleneck course subjects. There were 86 faculty participants from 19 campuses, recommended by their Provosts and other campus leaders. Faculty learned established pedagogical approaches and enabling technologies that have improved student success in bottleneck courses.

The eAcademies were organized very quickly so that CSU faculty could redesign courses over the 2013-2014 Academic Year. Participant evaluations validated the peer-led, collaborative approach taken in the eAcademies. They also highlighted diverse motivations for attending and suggested activities that would help participants in the yearlong redesign program. Overall, the evaluations demonstrated that the eAcademies had a deep and lasting impact on faculty and that this program could be expanded in future years to serve additional faculty throughout the CSU.

eAcademy and Participation by Faculty and  Campus

  • Engineering @ San José, July 22-24, 2013
    • 19 faculty from 13 campuses focused on edX model
  • Physics @ Pomona, July 29-31
    • 19 faculty from 5 campuses focused on smartPhysics
  • Chemistry @ Fullerton, July 31 – Aug 2
    • 11 faculty from 4 campuses focused on Supplemental Instruction
  • Mathematics @ Northridge, Aug 7-9
    • 21 faculty from 10 campuses focused on Supplemental Instruction
  • Critical Thinking @ Chico, August 12-14
    • 16 faculty from 9 campuses focused on fully online teaching
  • Overall: 86 faculty from 19 CSU campuses

Each eAcademy focused on the unique pedagogies and technologies used by the proven courses to address bottleneck issues. In addition, there were common topics at each eAcademy, such as the background and goals of the Proven Course Redesign program, engaging 21st Century Learners, aligning student learning outcomes with assessment, CSU’s Quality Online Learning and Teaching program, the Affordable Learning Solutions initiative, accessibility, early warning systems, learning analytics, and ongoing discussions that would take place in the online professional learning communities throughout the year.

Additional common themes included the “flipped classroom,” lecture capture/online videos, online homework, and supplemental instruction. After participating in the eAcademy, faculty participants were offered the opportunity to join an ongoing Chancellor’s Office program to conduct their own redesign to incorporate elements of the successful course.

This evaluation is based on the responses of 55 attendees (64% response rate). See Appendix A for a copy of the evaluation questions. Overall, faculty found the eAcademies rewarding and beneficial: 98% reported that attending the eAcademy was worthwhile, with comments such as ” . . .the workshop provided a lot of information on the new course materials and implementation method as well as a forum for future support.” Additional themes that emerged from the evaluations follow below.

Primary reasons for attending the eAcademy: Discover New Pedagogies/Technologies to Improve all Courses and Learn about the Redesign Model Courses*

bubbles why eacademyWhen asked “What is the primary reason you attended the eAcademy?” the largest proportion (49%) of faculty at the eAcademy came to learn new pedagogical strategies and technologies that they could immediately apply to their teaching, regardless of their participation in the ongoing redesign program.

The second highest number of respondents (31%) attended to learn about the proven course for potential adoption / adaptation in the full-year course redesign program. Collaboration and learning from peers were rated lower as motivating factors for attending the eAcademy.

*Note: faculty could only select one response from those provided, for all other questions, faculty could select multiple responses.

2. eAcademy Outcomes and Impact

Benefits from Attending the eAcademy: Share Experiences, Learn from Colleagues, Identify Elements of the Model Course” and Learn About Redesign

When asked to indicate their level of agreement with five statements about the eAcademy they attended, agreement with all of the statements exceeded 65%. The strongest agreement was with the statement “The Academy provided opportunities to share experiences and learn from colleagues” (98%), even though engaging with CSU colleagues and learning what they were doing was the primary purposes of attending for 9% of the faculty. Statements specific to the course redesign also achieved high levels of agreement, with 91%, 87%, and 80% agreeing with statements that the Academy helped them to identify the important elements of the model, helped them to understand expectations and requirements of the Course Redesign Program, and increased their awareness of broader pedagogy and technical practices, respectively. These benefits are not a surprise considering the primary purposes for attending identified by participants. Hands-on experience was rated lowest at 65%.

“What benefits did you achieve by attending the eAcademy?

Perceived eAcademy Benefits

Changing Teaching Practices as a Result of Attending the eAcademy: Replacing and Revising Course Activities and Seeking Affordable Learning Solutions

When asked which changes they hoped to make to their teaching practices (whether within or outside of the redesigned course) as a result of attending the eAcademy, the respondents rated highest their hope to replace and/or revise their course activities (40 respondents*), in strong alignment with their goal of discovering new pedagogical strategies. Their second highest response (24 respondents) indicated that they also planned to seek more Affordable Learning Solutions. Revising course materials, redesigning the overall approach and replacing/revising course assessments were selected by 20-21 of the participants. Both seeking more Affordable Learning Solutions and changing course assessments were added benefits from attendance. Least highly rated was replacement of their course materials (15 respondents); this is fairly closely aligned with the seventeen respondents who came to the eAcademy with the purpose of adopting or adapting the course model presented at the Academy.
*Because this question allowed multiple responses, the number of respondents, and not percentages, are used to indicate the frequency of responses.

“How will you change your teaching practices as a result of attending the eAcademy?”

change teaching practice

Strong Participation in the Yearlong Course Redesign Program by those Attending for that Purpose.

Forty-five percent of the respondents planned to participate in the full yearlong Course Redesign Program, with 42% not sure. Among those who identified their primary reason for attending the eAcademy as “To adopt/adapt this course model in the ‘Proven Course Redesign’ program,” 76% planned to participate, with 13% unsure. In other words, of those who attended with the purpose of using the model, a large majority planned to continue on that course with ongoing participation.

“Do you plan to participate in the year-long Course Redesign Program?”

yearlong redesign plans

Ongoing Support for Redesign Program: New Technologies & Pedagogies, Model Course Information, and Collaboration Opportunities.

Those who indicated that they planned to participate in the Year-long Course Redesign Program were asked to rate the importance of ways they could be supported in their work over the academic year. All of the options were ranked at 62% or higher for importance. The broadest level of interest was in “more information about best practices using technology for engaging students” and “more information about the model course.” (88% rated both as extremely important or important.) This kind of support builds on their initial reasons for joining the eAcademy. Also of high importance were “opportunities to network with colleagues through a Professional Learning Community”(81%). While many did not cite working with colleagues as a primary purpose, they remained consistent with seeing collegial interaction and networking as a benefit — and one they want to continue. Getting feedback on course materials and activities was ranked least important, but 62% still ranked this as extremely important or important. Given the high importance of all of these methods for continuing support, it is clear that building on the strategies explored during the eAcademy is essential to the full realization of this program as the beginning of curriculum transformation targeting student success.

“What support can we provide for your course redesign over the coming year?”

support desired

What made attending the eAcademy worthwhile for you? – Highlights

  • “The opportunity to learn new techniques and technologies for guiding students’ success, to collaborate with colleagues at various levels of experience with online teaching, to explore the possibilities for effective online teaching, to engage in debate about best practices, and to finally learn how to use Word to make accessible documents. All the while I was thinking about how to apply what we were learning to my own course, in the evening I went online to adjust my own materials in light of the day’s experiences. That’s pretty intense AND rewarding.”
  • “Basically, I learned a great deal about the direction the CSU means to take in the very near future. I was also encouraged to participate in the coming changes.”
  • “The academy provided an opportunity to see what other CSU faculty are doing to improve student success rates in mathematics courses that are traditionally difficult for students. “
  • “Learning techniques that have been both successful and not successful from colleagues. An environment that was honest and encouraging. “
  • “Two of my primary interests are teaching and learning and statistics. I was thrilled to be in an environment where we were exposed to research driven strategies are employed in teaching and learning. “
  • “An opportunity to see what colleagues are doing in the area of on-line critical thinking. Helped to confirm that the experiences I was having were not unique, which provides encouragement to keep trying.”

3. Conclusions, Lessons Learned, and Next Steps

The eAcademies provided an opportunity for faculty to exchange ideas, learn innovative practices, and evaluate new technology applications that have demonstrated success improve the achievement of CSU students. By learning from their colleagues, in collaboration with Chancellor’s Office staff, participants were able to take away specific techniques applicable for their discipline along with broadly applicable best practices that can be used across bottleneck courses.

In their comments, several participants mentioned that they thought the eAcademies could have a larger impact if they were more broadly promoted. During this first year of the program, given the short timetable, the eAcademies were not “advertised;” instead, campus Provosts were asked to recommend faculty participants. In future years, this approach could be rethought and perhaps a broader marketing approach could be taken that would increase participation and deepen the impact of the eAcademies.

For the faculty choosing to participate in the year-long redesign program, a Professional Learning Community has been created that includes twice-monthly web conferencing meetings on topics of interest, opportunities for interaction with fellow faculty redesigning courses, and an online “workspace” to contribute works in progress and provide feedback to other faculty in the program.