Ed Tech IMS Caliper Interoperability learning engineer learning engineering Tech

Learning Engineering: A Caliper Example –

Higher ed LMS market share for US and Canada, January 2019

In my current IMS replace submit, I wrote,

[T]he nature and challenges of interoperability our sector might be dealing with in the next decade are basically totally different from those that we faced in the last one. Up till now, we’ve got primarily been involved with synchronizing administration-related bits throughout purposes. Which individuals are on this class? Are they students or instructors? What grades did they get on which assignments? And how much does each task rely towards the ultimate course grade? These challenges are exhausting in all the methods which are acquainted to anyone who works on any kind of generic knowledge interoperability questions. 
But the next decade is goes to be about knowledge interoperability because it pertains to perception. Knowledge scientists assume that is nonetheless acquainted territory and are excited because it keeps them at the frontier of their very own career. But this won’t be generic knowledge science, for several reasons.

I then asserted the following positions:

  • As a result of learning processes are usually not immediately observable, blindly operating machine learning algorithms towards the press streams in our studying platforms will in all probability not train us a lot about studying.
  • Then again, if our analytics are theory-driven, i.e., if we start with some empirically grounded hypotheses about learning processes and design our analytics to seek for knowledge that either help or disprove these hypotheses, then we’d truly get someplace.
  • As a result of studying analytics expressions written in the IMS Caliper normal may be readily translated into plain English, Caliper might type a foundation for expressing instructional hypotheses and translating them into interoperable instruments for testing these hypotheses throughout the boundaries of tech instruments and platforms.
  • The type of Caliper-mediated conversation I imagined among studying scientists, working towards educators, knowledge scientists, learning system designers, and others, is related to a time period coined and still used closely at Carnegie Mellon University—”learning engineering.”

On this submit, I’ll discover the last two points in more element.

What the heck is “studying engineering”?

The term “learning engineering” was first used by Nobel laureate and Carnegie Mellon College polymath Herbert Simon in 1966. It has been round for fairly a while. But it’s a term whose time as lastly has come and, as such, we are seeing the standard educational turf wars over its which means and value. On the one hand, some people like it, embrace it, and need to apply it liberally. IEEE has a whole group devoted to defining it. As is all the time the case, a few of this type of enthusiasm is considerate, and a few of it’s less so. At its worst, there is a tendency for individuals to get snarled in the time period as a result of it offers a sure je ne sais quoi they have been yearning for to explain the elements of their jobs that they really need to be doing as change agents quite than the mundane tasks that they maintain being dragged back into doing, very similar to the best way some people are wrapping “innovation” and “design” round themselves like a heat blanket. It’s perfectly comprehensible, and I feel it attaches to something real in lots of instances, however it’s onerous to say precisely what that is. And, in fact, the place there are fanatics in academia, there are critics. Once more, some thoughtful, while others…less so. (Notice my remark within the thread on that notably egregious column.)

If you want to get a transparent sense of the vary of potential meanings of “learning engineering” as utilized by individuals who truly give it some thought deeply, one good place to start out can be Learning Engineering for Online Schooling: Theoretical Contexts and Design-Based mostly Examples edited by Chris Dede, John Richards, and Bror Saxberg. (I am still engaged on getting half a day’s value of Carnegie Mellon College video shows on their own learning engineering work ready for posting on the internet. I promise it’s coming.) There are lots of great take-aways from that anthology, one in every of which is that even the individuals who assume exhausting concerning the time period and work collectively to put collectively something like a coherent tome on the topic don’t absolutely agree on what the time period means.

And that’s actually OK. Let’s just set a couple of boundary circumstances. On the one hand, studying engineering isn’t an all-encompassing discipline and methodology that’s going to make all earlier roles, disciplines, and methodologies obsolete. In case you are an educational designer, or a studying designer, or a consumer expertise designer; in the event you apply design considering, or ADDIE; be not afraid. Then again, learning engineering just isn’t creeping Stalinism both. Think about studying engineering, writ giant, as applying knowledge and cognitive sciences to assist result in desired studying outcomes, often inside the context of a workforce of colleagues with totally different expertise all working collectively. That’s still fairly obscure, nevertheless it’s particular sufficient for the present cultural moment.

Overlook about your stereotypes of engineers and their practices. Do you consider there’s a place for applied science in our efforts to improve the ways by which we design and deliver our programs, or attempt to perceive and serve our students needs and objectives? In that case, what would such an utilized science appear to be? What would a person applying the science have to know? What would their position be? How would they work with different educators who’ve complementary expertise?

That’s the risk area that studying engineering inhabits.

Utilized science as a design train

One of many reasons that folks have hassle wrapping their heads around the notion of studying engineering is that it was conceived of by very uncommon mind. A number of the critiques I’ve seen online of the time period place “studying engineering” in opposition to “studying design.” However as Phil Lengthy points out in his essay in the aforementioned anthology, Herb Simon each coined the time period “studying engineering” and is actually the grandfather of design considering:

Design science was introduced by Buckminster Fuller in 1963, nevertheless it was Herbert Simon who’s most intently associated with it and has established how we consider it at present. “The Sciences of the Artificial” (Simon, 1967) distinguished the synthetic, or sensible sciences, from the pure sciences. Simon described design as an ill-structured drawback, very similar to the training surroundings, which includes man-made responses to the world. Design science is influenced by the restrictions of human cognition in contrast to mathematical models. Human decision-making is additional constrained by practical attributes of restricted time and out there info. This bounded rationality makes us susceptible to hunt enough versus optimal options to issues. That is, we interact in satisficing not optimizing. Design is central to the factitious sciences: ‘Everybody designs who devises programs of action aimed toward changing present conditions into desired ones.’ Natural sciences are involved with understanding what’s; design science as an alternative asks about what must be. this distinction separates the research of the science of learning from the design of studying. Learning scientists are fascinated about how humans study. Studying engineers are a part of group targeted on how college students should study.”

Phil Lengthy, “The Position of the Studying Engineer”

Phil factors out two essential dichotomies in Simon’s considering. The primary one: is vs. ought. Natural science is about what is, whereas design science exists about what you want to exist. What you need to deliver into being. The second dichotomy is about properly structured vs. poorly structured. For Simon, “design” is a set of actions one undertakes to unravel a poorly structured drawback. To wish or want is human, and to be human is to be messy. Understanding a human need is to know a messy drawback. Understanding how totally different people with totally different backgrounds and totally different cognitive and non-cognitive talents study, given a wide range of contextual variables just like the educating strategies being employed, the private relationships between college students and instructor, what else is occurring in the college students’ lives on the time, whether totally different students are coming to class properly fed and nicely slept, and so on, is just about the definition of a poorly structured drawback. So so far as Herb Simon is worried, schooling is a design drawback by definition, whether or not you select to make use of the phrase “engineer.”

Within the next part of his article, Phil then makes an interesting connection between the evolution of design considering, which emerged out design science, and studying engineering:

Design considering represents these processes that designers use to create new designs, attainable approaches to drawback solutions areas where none existed before. A problem-solving technique has been derived from this and applied to human social interactions iteratively taking the designer and/or co-design members from inspiration to ideation and then to implementation. The designer and design staff might have a psychological mannequin of the solution to a proposed drawback, however it’s important to externalize this representation when it comes to a sketch a description of a learning design sequence, or by actual prototyping of the actions which the learner is requested to interact. [Emphasis added.] All concerned can see the attributes of the proposed design answer that weren’t apparent in the conceptualization of it. this means of externalizing and prototyping design solutions permits it to be located in bigger and totally different contexts, what Donald Schon referred to as reframing the design, situating it in contexts aside from originally thought-about.

Phil Long, “The Position of the Learning Engineer”

So the important function that Phil is looking out in design considering is placing the thought out into the world so that everyone can see it, respond to it, and speak about it collectively. Now watch the place he takes this:

As learning environments are intentionally designed in digital contexts, the chance to instrument the training setting emerges. Learners benefit when it comes to suggestions or steered attainable actions. Evaluators can assess how the course carried out on quite a lot of dimensions. The school and others in the learning-design workforce can get knowledge by means of the instrumented studying behaviors, which can provide insight into how the design is working, for whom it’s working, and in what context.

Phil Long, “The Position of the Learning Engineer”

Slightly than a sketch, a wireframe, or a prototype, a studying engineer makes the graph, the dashboard, or the visualization into the externalization. For Herb Simon, as for Phil Long, these design artifacts serve the identical objective. They’re the identical thing, principally.

In case you’re not a knowledge individual, this could be onerous to understand. (I’m not a knowledge individual. This is exhausting for me to understand typically.) How are you going to take numbers in a table and turn them into a meaningful artifact that a group of individuals can take a look at together, talk about, make sense of, debate, and study from? What may that even appear to be?

Properly, it’d look something like this, for example:

Phil Hill’s well-known squid diagram

Phil Hill has a graduate diploma in engineering. Not learning engineering. Electrical. (Also, he is not a Stalinist.)

By the best way, once we we externalize and share knowledge with a scholar about her studying processes in a type that’s designed to impress thought and dialogue, we now have a specific term of artwork for that in schooling. It is referred to as “formative evaluation.” If we do it in a method such that the scholar all the time has access to such externalizations, that are regularly updating based mostly on the scholar’s actions, we call that “continuous formative evaluation.” When executed nicely, there’s proof that it can be an effective instructional follow.

Caliper statements as studying engineering artifacts

So this is the place we have arrived at this level in the publish:

  • Design is a process by which we deal with ill-defined issues of meeting human wants and needs, akin to needing or eager to study one thing.
  • Engineering is a word that we’re not going to fret about defining exactly for now, however it relates to applying science to a design drawback, and subsequently typically includes the measurement and numbers.
  • One essential innovation in design methodology is the creation of exterior artifacts early within the design process so that numerous stakeholders with differing types of experience and expertise can present suggestions in a social context. In different phrases, create one thing that makes the thought extra “real” and subsequently simpler to discuss.
  • Learning engineering consists of the talents of creation and manipulation of design artifacts that require extra technical expertise, together with expertise in knowledge and software program engineering.

The twist with Caliper is that, slightly than utilizing visualizations and dashboards because the externalization, we will use human language. This was the original concept of behind the Semantic Net, which continues to be sensible in concept, even if the original implementation was flawed. Let’s assessment that primary concept as carried out in Caliper:

  • You possibly can categorical statements concerning the world (or the world-wide net) in three-word sentences of the form [subject] [verb] [direct object] e.g., [student A] [correctly answers] .
  • As a result of English grammar works the best way it does, you’ll be able to string these sentences collectively to type inferences, e.g., [tests knowledge of] [multiplying fractions]; subsequently, [student A] [correctly answers] [a question about multiplying fractions].
  • We will define obligatory and optionally available particulars of every noun and verb e.g., it may be obligatory to know that question 13 was a a number of selection question, however it may be non-compulsory to include the precise text of the question, the right answer, and the distractors.

That is it. Three-word sentences, which work the best way they do in English grammar, and definitions of the “phrases.”

A learning engineer might use Caliper paragraphs as a design artifact to facilitate conversations about refining the usual, the products concerned, and the experimental design. I am going to share a modified model of an example I just lately shared with an IMS engineer for example this similar level.

Suppose you have an interest in helping college students grow to be better at reflective writing. You need to do this by providing them with continuous formative assessment, i.e., along with the suggestions that you simply give them as an teacher, you need to provide them an externalization of the language of their reflective writing assignments. You need to use textual evaluation to assist the scholars take a look at their very own writing by way of a new lens, discover the spots where they’re really doing critical thought work, and in addition the spots the place perhaps they might assume somewhat more durable.

But you need to clear up a number of problems as a way to do give this affordance to your students. First, it’s a must to develop the pure language analysis software that may detect cues in the college students’ writing that indicate self-reflection (or not). That is exhausting sufficient, but the analysis is being carried out and progress is being made. The second drawback is that you’re designing a brand new experiment to test your latest iteration and wish some type of summative measure to test towards. So perhaps you design a randomized managed trial the place half the scholars in the class use the new feedback software, half don’t, and all get the identical human-graded last reflective writing task. You examine the results.

This is an example of theory-driven studying analytics. Your principle is that scholar reflection improves when students grow to be more conscious of sure kinds of reflective language of their journaling. You assume you’ll be able to practice a textual analysis algorithm to reliably distinguish—externalize—the type of language that you really want college students to be more aware of of their writing and level it out to them. You need to check that by giving students such a device and see if their reflective writing does, the truth is, enhance. Both college students’ reflective writing will improve underneath the check condition, which can provide supporting evidence for the idea, or it will not, which at the very least won’t help the idea and may present evidence that tends to disprove the idea, depending on the specifics. There are knowledge science and machine studying being employed here, however they’re being employed extra selectively than just shotgunning an algorithm at a knowledge set and expecting it to provide you with novel insights concerning the mysteries of human cognition.

Setting up theory-driven learning analytics of the type described here is difficult enough to do in a unified system that’s designed for the experiment. But now we get to the problem for which we’ll want the assistance of IMS over the subsequent decade, which is that the varied actions we need to monitor for this work typically occur in several purposes. Every writing task is in response to a reading. So the first thing you may need to do, a minimum of for the experiment if not in the manufacturing software, is to regulate for college kids who do the studying. If they are not doing the studying, then their reflective writing on that reading is not going to inform you a lot. As an example the reading occurs to put in an e-book app. However their writing takes place in a separate pocket book app. Perhaps it’s no matter pocket book app they usually use—Evernote, One Observe, and so on. Ideally, you’d want them to journal in no matter they normally use for that type of exercise. And if it’s reflective writing for their very own progress, it ought to be an app that they own and that may travel with them after they depart the class and the institution. Then again, the ultimate writing task needs to be submittable, gradable, and perhaps markable. So perhaps it will get submitted via an LMS, or perhaps via a specialised software like Turnitin.

That is an interoperability drawback. Nevertheless it’s a particular one, as a result of the semantics should be preserved via all of those connections to ensure that (a) the researchers to conduct the research, after which (b) the formative assessment device to have actual worth to the scholars. The individuals who normally write Caliper metric profiles—the technical definitions of the nouns in Caliper—would do not know about any of this on their own. Nor would the appliance builders. Each teams would wish to have a conversation with the researchers with a purpose to get the clarity they need in an effort to outline the profiles for this objective.

The language of Caliper might assist with this if a person with the best position and experience have been facilitating the conversation. That individual would begin by eliciting a set of three-word sentences from the researchers. What do you want to know? The answers may embrace statements like the following:

  • Scholar A reads text 1
  • Scholar A writes text alpha
  • Textual content alpha is a learning reflection of textual content 1
  • Scholar A reads textual content 2
  • Text 2 is a studying reflection of texts 1 and a couple of
  • And so on.

The individual asking the questions of the researcher and the function designer—let’s name that individual the training engineer—would then ask questions concerning the meanings and details of the phrases, akin to the following:

  • In what system or methods is the studying activity occurring?
  • Do it is advisable know if the scholar started the reading? Finished it? Anything finer grained than that?
  • What do you should know concerning the scholar’s writing so as to perform your textual analysis? What knowledge and metadata do you want? And the way long a writing pattern do it is advisable to elicit with a purpose to perform the type of textual evaluation you plan and get worthwhile results back?
  • What do you imply if you say that text 2 is a reflection of both textual content 1 and a couple of, and the way would you make that willpower?

Sooner or later, the info scientist and software techniques engineers would be a part of in the conversation and totally different considerations would start to come up, similar to the following:

  • Right now, I’ve no method of associating Scholar A in the note-taking system with Scholar A in the reading system.
  • To do the analysis you need, you want the complete textual content of the reflection. That’s not presently within the spec, and it has performance implications. We should always talk about this.
  • The scholar knowledge privateness implications are very totally different for an IRB-approved analysis research, an individual scholar dashboard, and an instructor- or administrator-facing dashboard. Who owns these privateness considerations and how can we anticipate them to be handled?

Discover that the Caliper language has turn out to be the externalization that we manipulate socially in the design exercise. There are two points of Caliper that make this work: (1) the three-word sentences are linguistically generative, i.e., they will categorical new ideas that have never been expressed earlier than, and (2) every human-readable expression immediately maps to a machine-readable expression. These two properties together allow rich conversations amongst very totally different kinds of stakeholders to map out theory-driven analytics and the interoperability requirements that they entail.

That is the type of conversation by which Caliper can evolve into a regular that leads to useful insights and tools for enhancing learning impression. And in the early days, it should possible happen one use case at a time. Over time, the working group would study from having sufficient of those conversations that design patterns would emerge, each for writing new parts of the specification itself and for the process by which the specification is modified and extended.

Copyright Carnegie Mellon College, CC-BY