The following sections contain the most frequently answered questions regarding the CSU Quality Assurance program. Clicking on each section header will reveal a series of common questions. Clicking on any of these questions will provide the official answers.
CSU QLT Program Background Info
Who developed QLT?
QLT was developed as a project within California State University, Academic Technology Services. The lead for the QLT project has been Brett Christie, who has extensive experience in teaching and learning effectiveness, faculty development, Universal Design for Learning, and instructional materials accessibility. In addition, QLT was informed by an extensive community of CSU faculty, instructional developers, faculty developers, and directors of academic technology.
When was QLT developed?
QLT was developed in Fall of 2011, extensively revised over the first year, and continually improved.
Is QLT completely original work or was it influenced by prior instruments?
QLT was developed after extensive research into existing instruments that attempted to measure and inform the quality of online courses. In addition, QLT was influenced by long-standing research on effective practices, both in traditional and online formats. Finally, QLT was influenced by extensive CSU efforts in the area of Universal Design for Learning and accessibility.
Which instruments most influenced QLT?
Given that QLT was developed as a CSU resource, it was heavily influenced by the Rubric for Online Instruction, developed at CSU, Chico in 2004 and adopted by over 100 institutions of higher education. In addition, QLT was significantly influenced by the Quality Matters rubric, which was developed in 2003-2007 through funding provided by the U.S. Department of Education. This open resource instrument was a significant influence in the development of the original QLT instrument.
How similar is the QLT instrument to the Chico Rubric for Online Instruction?
While the Chico ROI was a well-developed and relatively successful rubric, it contained measurement elements that were fewer in number and broader than desired going forward. QLT has significantly more objectives that offer a more varied and discreet course effectiveness analysis.
How similar is the QLT instrument to the Quality Matters rubric?
The QLT instrument contains 57 objectives, while the QM rubric contains 43 objectives. The main difference is that QM only focuses on delivery, while QLT focuses on design and delivery. Therefore, QOLT has 9 objectives that are not represented in QM. For further information, see the QOLT-QM Alignment Matrix.
What instrument is part of the CSU QLT program?
The CSU QLT Informal Self-Peer Review instrument allows instructors to perform a self-rating of their course, and for peer-reviewers (e.g., colleague, instructional developer) to provide feedback. Those completing the CSU QLT self-rating instrument can have a copy of the results sent to them by electing to do so at the end of the instrument. We are not responsible for maintaining a database or providing results on demand.
What is meant by “QOLT Awards and Recognition program” instruments?
The QLT instrument used for the official CSU QLT Course Certification program runs during the academic year and invites faculty who would like to self-rate their blended/hybrid or online course and submit for further review at the campus level. In addition, the QLT program includes a student-ratings instrument.
What is meant by “QLT Informal” instruments?
The CSU QLT instrument may be completed at any time and are for informal faculty self-rating. Instructors can choose to complete the QLT self-rating instrument at any time and simply have the results sent to their via email. They may also choose to have a copy sent to a colleague or instructional staff member who may be assisting them with course redesign. The data go into a secure database but the data are not managed, reported on, or sent to others.
Do I need to complete the QLT instrument in one session?
No. We have enabled a Save and Continue feature within each page of the survey. If you choose to leave the survey and return later, you may click on Save and Continue. You will be sent an email with a link for you to continue the survey later.
When I complete the QLT instructor self-rating instrument, how do I get the results?
At the end of the survey, you can choose to have a copy of the completed survey sent to you via email. In addition, you have the option of sending it to someone else (e.g., QLT Campus Coordinator or instructional developer with whom you may be working).
QLT Awards and Recognition Program
What is the QLT Awards and Recognition Program?
The program was developed as a way to raise awareness toward effective online teaching and learning. The approach has been to both inform and acknowledge exemplary practice.
Sharing of Quality Assurance Exemplars
Are any of the CSU QA exemplary courses and/or course elements shared?
Yes. The Quality Assurance Resource Repository (QuARRy) is a collection of teaching-learning exemplars collected from participating CSU campuses. These exemplars are indexed according to both the CSU QLT instrument and the Quality Matters rubric. This allows faculty, administrators, and instructional developers to view multiple ways to effectively meet a particular QLT/QM objective. Our hope is that it will continue to grow and foster collaboration among CSU campuses. Here is a link to a 1-page flyer about the repository and access.
How do I access the Quality Assurance resource repository?
- Go to http://quarry.calstate.edu
- Use the browsing tools on the right to explore according to QLT/QM objective or enter search terms.
Will the Quality Assurance resource repository continue to expand?
Yes. Our hope is that the repository will continue to grow and foster collaboration among CSU campuses as faculty continue to submit exemplary course components.
How does one contribute to the QA resource repository?
There is an online submission form at the QuARRy site.
What intellectual property rights are associated with resources that are posted to the repository?
As part of the submission process, we ask the resource owner to choose a Creative Commons license to be assigned to the resource. These licenses allow the contributor to clearly indicate the level of sharing and repurposing they would like to allow, while limiting any commercial use of their resource. Any resource submitted to the QLT online resource repository will be branded with the author’s chosen Creative Commons license.
QLT Data Collection and Storage
In what ways is data collected through the QLT program?
QLT has multiple instruments and stages of data collection. However, there are two distinct sets of instruments and data collection processes: QOLT Awards and Recognition program instruments, and; QOLT Non-awards instruments (link). These are further described below.
Where are the QLT data stored?
The data submitted through all QLT instruments reside in a central database stored off-site through our SurveyMonkey Pro subscription. These data are backed up regularly and protected.
Who has access to QLT data?
The only person who has access to the QLT database is an authorized CSU Academic Technology Services staff member who is our SurveyMonkey Pro instrument developer and data manager. Access to the QLT database is off limits to all others and data-related inquiries can only be made through Brett Christie, QLT Director.
Who may request QLT data and what information is provided?
The only persons authorized to request and receive data are designated QLT Campus Coordinators, who are serving to help promote and manage the QLT Awards and Recognition program. These coordinators are given the data for sharing a copy to the instructor only, as well as performing an authorized peer-review of the course the instructor has submitted for further review. No data from the instructor self-rating or peer-review are shared beyond the instructor and QLT Campus Coordinator.
What reports are created from the QLT database?
Annual reports are developed using aggregate data only. No faculty, staff, or student names are included. Reports are simply made as to how many faculty participants, student-ratings submitted, average total QLT score across faculty, average total rating given by students, mean rating per objective across faculty group. By analyzing mean ratings per objective across faculty, it is possible to identify possible strengths and weaknesses common to teaching blended-online courses. Professional development can be offered toward any of the common weakness trends observed.
Are QLT data used for faculty personnel review purposes?
QLT data are absolutely not intended to be part of the faculty personnel review process (e.g., evaluation of teaching effectiveness), which is to be respected as the official method of determining faculty teaching performance. The QLT data collected and any consultation that may take place around them are strictly for voluntary faculty exploration and development purposes. If faculty choose to discuss their experience using QLT to make their courses more effective within their statement of teaching effectiveness document, or similar, that is their choice. Only individual faculty are authorized to make any statements in their review regarding QLT and any of the data they may have been provided. QLT data are not provided by CSU QLT personnel to any individuals other than the official QLT Campus Coordinator.
About QLT/QM Course Reviews
The process involves other faculty looking at my course to make recommendations. Are there requirements for confidentiality?
Who are the members of a course review team?
What are the eligibility requirements to be a course peer reviewer?
- Successful completion of Reviewing Courses Using the QLT Instrument or Applying the Quality Matters Rubric workshops
- Successful completion of the Peer Reviewer Course
- Have current online teaching experience (within the last 18 months)
- Must have a signed MOU on file with QM (if serving as reviewer toward QM course certification)
Will the review teams for CSU campus courses be made up of only Peer Reviewers from other CSU campuses?
About Course Review Roles & Responsibilities
What’s the difference between a Peer Reviewer and a Master Reviewer?
Peer Reviewers are experienced online faculty members who have completed the Peer Review Certification course. The Master Reviewer is an experienced Peer Reviewer who has completed additional training. The Master Reviewer has additional responsibilities during an official Course Review, including serving as the team chair. They are responsible for facilitating the review, all team communications, and serve as the authoritative guide on matters related to application of the QM Rubric or QLT instrument and process.
What’s the difference between a Systemwide Subscription and an Affiliate Subscription to Quality Matters?
CSU has a Systemwide Subscription to Quality Matters. This subscription covers all 23 CSU campuses. Each of the campuses is thereby offered the opportunity to become an affiliate subscriber under this systemwide subscription. As part of this affiliate subscription, campuses have identified a campus Institutional Representative to help coordinate the campus QM activities. Campus QM IRs are expected to work through the Systemwide QM IR (Ashley Skyler) for acquisition of training or course reviews.
About Professional Development Opportunities
What QA trainings are offered in the CSU?
The following workshops are offered within the CSU at a cost of $25 each and occur in both 3-week online asynchronous format and one-day face-to-face format.
Introduction to Teaching Online Using the QLT Instrument: This course is an introduction to teaching online, with the CSU QLT instrument embedded across 9 modules representing the 9 sections of QLT. This course is intended for anyone interested in learning how to teach online.
Reviewing Courses Using the QLT Instrument: This course provides an in-depth experience with the QLT instrument and how to use objective-based examples to support reviewing and informing blended-online courses. Participants will engage in hands-on experiences using the QLT objectives to rate elements of a sample course, learning how to write helpful recommendations, and discussing examples of setting up a peer-review process on their campus. This session is for those looking to use the instrument for a self- and/or peer-review.
Applying the QM Rubric: An overview of Quality Matters rubric and annotations, the research underlying the QM rubric, and the peer-review process for evaluating the quality of online course design. Participants will engage in hands-on experiences applying the QM standards to a sample course, writing helpful recommendations, and ensuring alignment between course level objectives, module level objectives, assessments, materials, activities, and tools.
What workshops and courses are available to CSU faculty/staff at a reduced cost through Quality Matters?
CSU faculty/staff at QM affiliate campuses are able to enroll in workshops offered by Quality Matters at a one-third cost. For example, many of the QM workshops are $300 but employees of CSU QM affiliate campuses pay only $100.
What is the difference, if any, between the online workshops and the 1-day on-site workshops?
If you’re trying to decide whether the online or F2F workshop is best for you:
Online Workshops – These asynchronous workshops are delivered over the course of 2-3 weeks by certified facilitators. In these workshops participants will: (a) Read short lesson materials related to the QM rubric and review process, (b) Take a few short, 10-question quizzes on the material, (c) Apply the QM rubric to a sample online course, (d) Participate in forum discussions related to each of the 8 General Standards, and (e) Apply the QM rubric to a design scenario. It provides an excellent opportunity for faculty to individually consider how the rubric applies to each standard in course development and how the review process works in a course review, exchange ideas and perspectives on course design with colleagues, and gain personal experience applying the rubric.
One-Day On-Site Workshops – These workshops are delivered by certified facilitators over the course of six hours. In this workshop participants will: (a) Learn about the QM/QLT rubric and review process through presentation materials and activities provided by 1-2 workshop facilitators, (b) Take one quiz on material presented, (c) Collaboratively apply the QM/QLT rubric to a sample online course, and (d) Apply the QM rubric to a design scenario. It provides an excellent opportunity for faculty to gain a general, introductory understanding of the QM rubric and how it might be applied in course development and in the course review process and exchange ideas and perspectives on course design with colleagues.
How should I decide whether to take the online or on-site workshop?
The online workshop is recommended for faculty who intend to use the rubric to design their own courses, may eventually submit a course for a QLT/QM course review in the future, or may decide to become a peer reviewer at some point in the future. This workshop provides an opportunity to look carefully at a sample course and make personal decisions about whether or not the standards are met, share your findings and reasons for your decisions with colleagues through the discussion threads, receive new ideas and perspectives from colleagues and write recommendations for improvement over the course of the two-week period. You can work at your own pace but will want to work and post along with colleagues to gain the most benefit from the course.
The face-to-face workshop is appropriate for faculty who are interested in learning about the rubric and review process from a more “overview” perspective. Due to the limited time period, course material is presented by a facilitator and while there is time for some discussion, due to time-constraints there is limited opportunity to individually practice applying the standards to a sample course.
When are the workshops offered?
A complete calendar of all CSU QA workshops offered through CSU ATS can be found here.