Learning Data Is Incompetent: Refocusing Education Measurement

Part 2 of Our 4 Part Series on Big Data in Learning

This series is extracted from the CogBooks white paper, “Big Data.” You’ll link to the next post at the end of each article. You can also use this menu to read the articles in the order you prefer.

Part 1: Big Data in Learning: The Emerging Value of Online Learning Datasets
Part 3: Ten Level Taxonomy of Data: Potential Sources of Learner Insights
Part 4: Flipped Statistics: The Changing Paradigm of Education Data

Let’s start with a dose of reality: education and training have always coveted data. But in any honest appraisal of this data collection, we have to admit that it is largely the wrong data. There has historically been too much focus on start and end point data. All dull inputs and outputs. What we need to focus on is the cognitive improvement of the learner. Here are five examples of data, mostly superficial, that account for the vast bulk of the data collected in education and training:

1. Bums in Seats

To measure attendees or bums in seats is to measure the wrong end of the learner. Yet this is what so much ‘contact time’ is in our colleges and universities. I once attended a talk by the Head of Training for a global bank, where she proudly showed that X number of meals had been served in her canteen on the training campus. And they wonder why banks failed?

2. Contact Time

Contact time is essentially an excuse for not measuring what is learned. Turning up is hardly a measure of learning. Attendance is not attainment. In some cases, the contact time is even more illusory, as, in Higher Education in the UK, they do not even count the number of students that turn up for lectures.

3. Course Completion

Completion is not a measurement of attainment or competence, yet so many courses measure simply this. We have already seen how turning up is not a great measure, but this is so often simply a measure of how many people just hung about until the end.

4. Summative Assessments

The problem with final test and exam data is that it’s all too late. The deed is done. Exams are too often the final act in learning and an end-point. This final mark so often stops even the best learners from trying any harder and marks the poorer students out as failures. There’s also the problem of cramming and short-term memory.

5. Happy Sheets

The evaluation of education and training is plagued with end-point data. None more futile than the obsession with post-course satisfaction evaluations, as they measure nothing. It’s a staple of classroom courses and often the only data that is collected. Yet, it says nothing about what has actually been learned.

Missing Data

What’s so often missing is the data on competence. We teach what is easy to test and test what is easy to teach. That means lots of academic knowledge which is tested through paper tests, from multiple choice to essays. The actual competence measured is often just the ability to cram and remember data to pass tests, quickly forgotten.

What really matters is data collected from learners as they learn. This is when data really is needed so that we can help learners succeed.

The lesson is to measure as directly as you can those things you wish to improve directly as part of the process (skills and competence), not through the abstract process of end-point testing. Summative assessments may not just be too late, they may also be too early, as they often test short-term recall, and even when applied, the application is in an artificial environment. Transfer is often missing. Most data collection in education and training, therefore, skates over the surface with data about superficial attendance, end-point assessment, and opinion. What’s missing is hard data on actual performance, competencies, and retained knowledge. What really matters is data collected from learners as they learn. This is when data really is needed so that we can help learners succeed. So much data focuses on the deficit model in education—failure and drop-outs.

Data and Adaptive Learning

We should also remember that data when it is not ‘big’ enough to be used with Google-like brute force is more likely to be used with traditional conditional branching, rule-sets, and algorithms, to be effective as a feed into adaptive, personalized learning. If you don’t have a large enough data set to cope with the messiness of the data and high-probability pattern matching, then you employ additional math to select and interpret the data.

Why Is Data, When Used by Adaptive Learning, So Important?

First, students need improved learning effectiveness through personalized support and reduced learning times through personalized pathways. Instructors need enhanced data and tools for helping students and increased automation of teaching tasks. In addition, University/Organizational Administrators need higher completion rates, reduced delivery costs, and increased use of data-driven methods.

When Is the Data Gathered and Recommendations Made?

Data may be gathered to improve formative feedback to improve the learner experience. This data can be used to accelerate learning by personalizing feedback screen by screen in online learning. It must be clear what data is gathered and then applied within the software. One can move beyond this to aggregated data by students who have taken that course and then feed that data back into the adaptive process to further improve the learning process. Course data can also be collected to improve the course content and structure, as well as drop-out and success rates.

What Data Is Used To Drive Adaptation?

General profile data such as educational background, demographic, and psychological data can be used, but it is prior educational achievement, activity, assessment, and instructor input that gives real leverage within an adaptive environment.

What Is Adapted?

One must be clear as to what is meant by ‘adapted’ in terms of adaption within a course. Data-driven sequencing is the goal, with content delivered in response to data-driven algorithms that lead to faster, targeted, improved mastery.

What Methods Drive Adaptation?

Data that is used by algorithms will drive efficient, flexible, and topic agnostic learning. This takes good design but uses data to target the right learning experiences for each different learner.

What Are Some End-Use Applications?

There are many possible adaptive contexts, such as flashcards, simulations, labs, assessments, e-learning courses, and blended learning.

How Extensive and Scalable?

Are we dealing with a component, course, or curriculum, and is the data gathered subject-specific or content-agnostic?

How Open Is the Content Model?

Content may be highly structured and closed within a proprietary system through to systems that are porous, and allow external content to be used in courses, even courses where much of the content is user-generated. Here, it is important to understand what data will be gathered and how.

How Accessible Are the Data and Architecture?

Sharable Content Object Reference Model (SCORM) has been a de facto standard for some time, allowing data to be extracted and used with interoperability in mind. Newer standards, such as TinCan, are emerging. Understanding what access one has to this data is important in terms of privacy and use.

Read Part Three: “Ten Level Taxonomy of Data: Potential Sources of Learner Insights”

Give Students Greater Agency and Instructors More Control

CogBooks weaves student agency, instructor empowerment, and curriculum affordability ($39.95 per course) into a comprehensive, adaptive learning platform. This simple to adopt and manage tool is a direct-replacement for textbooks. Higher education institutions or instructors can choose CogBooks for a single course or create an entire degree program such as the Biospine Initiative at Arizona State University. The CogBooks adaptive learning platform has been used by more than 200,000 students worldwide. It is proven to reduce dropouts by 90%* while improving student performance by 24%.* Connect with us if you’re interested in learning more, creating a custom course, or developing an entire degree program.

*Data from a consecutive four-year study in Introduction to Biology for Non-Majors at Arizona State University.