Academics & Academia Bits & Bytes course design Kevin Kelly Pedagogy quality matters Research rubric Tech

Online Course Design Rubrics, Part 1: What are they? –

Over the previous few years, the staff at e-Literate has reported on online course quality and common challenges online learners face. In late 2016, Phil Hill described and shared “explainer movies” that outlined how system-wide online course exchanges and their shared social infrastructure will help improve entry and improve quality. Virtually a yr ago Phil carried out an off-the-cuff, inner assessment of two online courses at Rio Salado School, during which he reviewed online course high quality in mild of the Making Digital Studying Work report by Arizona State University and the Boston Consulting Group. In a newer (February 12) article, Phil discussed online schooling challenges and potential options—specifically, 1) group school college students face a variety of challenges in on-line course environments and a couple of) establishments handle some of these challenges by way of online course design efforts at scale.

At present, the primary technique to scale on-line course high quality is thru using rubrics that inform on-line course (re)design. Understanding the rubric subject is crucial for instructional establishments that a) need to supply quality online packages, credentials, and/or certificates, and b) need to improve online scholar success charges.  To help these institutions—especially those that don’t yet use a rubric at scale—as well as the rubric providers themselves, I carried out a assessment of probably the most extensively used on-line course design rubrics, which I’ve became a three-part collection:

  • Part 1: WHAT? A comparison of the seven most widely used on-line course design rubrics, along with their collective strengths and limitations
  • Part 2: SO WHAT? A dialogue of why utilizing these rubrics has grow to be so essential, and some early proof of impression
  • Half three: NOW WHAT? Suggestions for what the rubric providers and adopters ought to do next to increase online scholar success further

Part 1 of three: WHAT?


My investigation of on-line course design rubrics began with publicity to two rubrics on the similar time:

I additionally knew about Quality Matters and needed to discover the complete vary of rubrics used to help high quality at scale. Curiosity did not kill any cats, however it led to look results itemizing several online course design rubrics*, each of which is utilized by a number of institutions (see Table 1).

TABLE 1. On-line course design rubrics utilized by a number of establishments

* NOTE: New Mexico State University also created a rubric that at one time was referenced by different faculties but has since switched to Quality Matters.

Velocity of change

Whereas a few of these online course design rubrics date again twenty years, their promotion and use at totally different levels—institution, district and system—really has ramped up in the final 5 years. To emphasise this level, think about the following. Baldwin, Ching and Hsu (2018) in contrast six of the seven rubrics listed in Table 1 above. 5 of those six rubrics have been up to date since these authors accomplished their research in July 2017. [NOTE: Ironically, the rubric comparison study did not appear in any of my search results. I only found it as a link from the OSCQR Resources.]

Relying on what milestone you select as a starting point, individuals have been finishing online courses for thirty years or so. For those of you wondering why we’ve got not gotten online course high quality found out by now, understand that face-to-face programs in larger schooling began over 1000 years ago and up to now don’t bear as a lot evaluation.


A lot of the seven rubric providers cite the Rubric for On-line Instruction (created by California State College, Chico) and some credit score the Quality Issues framework as influences used to create their very own rubrics. Some additionally cite Universal Design for Studying rules, the Group of Inquiry framework, and the National Survey of Scholar Engagement. Apparently, multiple rubric cites as an affect the Seven Rules for Good Apply in Undergraduate Schooling outlined by Chickering and Gamson (1987), which was revealed just earlier than the primary online programs launched. Virtually thirty years later, Crews, Wilkinson, and Neill (2015) applied these seven rules to on-line programs.

What the rubrics evaluate

Each rubric is damaged into broad classes—starting from three to ten, depending on the rubric. Across all seven rubrics there are fourteen classes in all, certainly one of which I’ve created. For this comparison, I reorganized every rubric to put comparable criteria in the same categories for consistency, and repeated criteria that measure multiple facet of an internet course. Desk 2 exhibits how many criteria each rubric consists of in every comparison class. (Daring numbers signify a rubric has 7 or more criteria in that class; italicized numbers signify a rubric has 2 or fewer standards in that class.)

TABLE 2. Number of standards each rubric consists of per comparison class  

Course Overview and Info 5 eight 10 9 9 10 four
Studying Goals 4 3 1 1 5 2 2
Educational Design & Course Supplies 4 9 5 7 6 12 10
Particular person Studying Actions 1 2 2 zero 0 1 0
Collaboration and Interplay 3 10 4 6 4 13 7
Facilitation 1 0 zero 7 0 zero 12
Evaluation 9 12 9 7 7 23 7
Learner Help 3 1 1 5 three three 2
Accessibility, Usability, Common Design, & Inclusivity 17 9 13 6 6 6 5
Course Abstract 0 1 0 3 0 zero 4
Course Analysis zero 0 zero zero zero zero 3
Course Know-how 2 6 four four four 7 four
Net Design or Course Format 0 0 four 0 zero 7 1
Cellular Platform Readiness zero zero 0 4 0 0 zero

Quantitatively, this enables us to see what elements of online educating every rubric emphasizes, which categories are nicely coated by each rubric, and which classes are usually not emphasised enough general. For example, all seven rubrics have a number of criteria associated to assessment, but solely two rubrics—QLT by the CSU system and OCE by UW LaCrosse—tackle an internet teacher’s facilitation methods. Similarly, all seven rubrics have several criteria related to accessibility, but on the whole they pay little attention to individualized studying actions.

To what extent does research help rubric criteria?

Past what number of criteria each rubric provides in a selected class, though, how nicely does studying design analysis and studying science help usage of those rubrics? Since MindWires leads the Empirical Educator Challenge—an effort to advertise broader adoption of evidence-based educating practices—I went by way of a variety of research articles and research literature evaluations. My objective was to determine correlations between a) the mixture set of rubric standards and b) online course design elements (or on-line educating methods) that have increased scholar engagement, motivation, persistence, success, and so on. Under is a consultant pattern of what I discovered.

  • The CCC Chancellor’s Office (2013, p. 23) identified elements that affect scholar persistence in online courses, which I’ve ranked in response to the amount of instructor affect to facilitate (from most influence to least affect):
    • Increased communication with teacher
    • Sense of belonging to studying group
    • Scholar satisfaction with on-line learning
    • Time management expertise
    • Peer/family help
  • Quite a lot of analysis articles (CCCCO, 2013; Crews, Wilkinson, & Neill, 2015; Hart, 2012; Nash, 2009; Orso & Doolittle, 2012; Ragan, n.d.; Savery, 2005) present that the web teacher performs one of the largest roles in scholar retention and/or success. Two examples embrace:
    • The CCC Chancellor’s Workplace (2013) identified faculty-student interplay and increased communication with the trainer as key elements in scholar success.
    • Communication/availability rated as the top attribute of an impressive online instructor, adopted by compassion, organization and suggestions (Orso & Doolittle, 2012).
  • In addition to interaction/communication with the web instructor, Lister (2014) shared the outcomes of a complete literature assessment figuring out key elements that have an effect on online learner success:
    • Course group and construction
    • Content material presentation
    • Opportunities for collaboration and interaction
    • Timely and effective instructor feedback

In Table 3 under, I’ve started the process of aligning analysis articles with rubric criteria in widespread rubric classes. While the research supports the concepts, though, not each rubric truly addresses them to the extent that the analysis dictates. For instance, 4 of the seven rubrics have a criterion associated to well timed teacher suggestions, but two of the four stop brief at simply sharing expectations for well timed suggestions. Those rubrics do not measure whether or not or not school truly give timely suggestions.

TABLE three. Analysis literature supporting rubric criteria 

Widespread rubric classes Research literature supporting rubric standards in those categories
Educational Design & Course Materials Ragan, Orso & Doolittle, eight studies cited by Lister
Interplay & Collaboration CCCCO, Crews et al., 9 studies cited by Lister
Evaluation Suggestions: CCCCO, Lister, Crews et al., Orso & Doolittle, Hart, & extra
Learner help CCCCO, Crawley, & extra

Some rubric suppliers also allude to or immediately reference different literature that supports the inclusion of rubric standards. For example:

On the flip aspect, the analysis identifies practices that the rubrics don’t handle a lot or in any respect. For example, a number of research show increased success when institutions require students to participate in a course-level orientation or module-based introduction to on-line learning (Cintrón & Lang, 2012; Lorenzi, MacKeogh & Fox, 2004; Lynch, 2001). Nevertheless, there are only a few standards across the rubrics that handle helping college students assess and handle their online learning readiness.

Limitations and strengths

As far as limitations go, all the rubrics focus heavily on reviewing courses before they start—i.e., before the students show up. Whereas all seven rubrics also have standards associated to interplay, those criteria largely handle activity set up, clarity of exercise directions, and different elements that can be reviewed without the students being in the course. Solely two of the seven rubrics—CSU QLT and UWL OCE—actually take a look at instructor facilitation.

Given these limitations, the rubrics do provide great value. Finally, these rubrics characterize the present eager about enhancing the quality of on-line programs. A few of the rubrics have advanced over twenty years and can continue to take action. The rubrics’ strengths manifest at totally different ranges—for individual instructors, a rubric acts as a course design guide; for establishments and methods, a rubric creates a standard vocabulary, an aspirational worldview, a mechanism for consistency and accountability, and the idea for a social infrastructure that runs parallel to the technological infrastructure.

In Part 2 of this collection, SO WHAT?, I will take a look at why these rubrics are essential for enhancing on-line course quality at scale, and how nicely rubric providers have evaluated the effectiveness of their instruments.

03/20/2019 UPDATE – Desk 1: Modified Rubric Provider Sort for Quality Matters to Non-profit group.

References for citations on this three-part collection