Adaptive Learning

LLLab Pro is a laboratory for creating new educational approaches and methods for lifelong learning

Adaptive Learning

How does it work
Goals and target audience
Теоретическое обоснование
Описание методики
Do's and don'ts
Дизайн тестирования
Команда проекта
Обратная связь

HOW TO WORK WITH TOOLS?

Look at the main concepts and theories.
All tools are grounded in theory and evidence. Armed with the understanding of the main concepts, you will know not only what to do, but also how to do it and why.

Take action!
Examine target audience, goals, learning outcomes and descriptions of tools. Develop a step-by-step plan to introduce a tool into your practice and identify the metrics which reflect each method's effectiveness.

Экспериментируйте!
К каждой методике мы добавили дизайн исследования, то есть вы сами можете оценить ее эффективность и написать нам результаты.
Присоединяйтесь таким образом

Experiment!
Treat all methods like hypotheses that need to be tested. For every tool we added a design of an experiment and metrics of success, so that you can share your results with us in a feedback form. Join our community!

1

2

3

Goals and target audience

If done right, adaptive learning can help achieve the following:

Save student's time—they learn all things they need faster without wasting time on aspects that are already familiar.

Help learners to finish the course—students are more satisfied with their progress, which brings down the dropout rate.

Read more here.

Target audience and learning solutions that can benefit from adaptive learning.

We recommend adaptive learning, if you have:

Defined subject area that does not change twice a year
Access to an automated learning system or resources to develop such a system
Many users that you are willing to educate (thousands or more)

As of today, there is not enough data to suggest that adaptive learning works only for a specific category of users (based on demographic factors or job responsibilities). Adaptive learning is a relatively new concept which is applied in a limited number of areas, such as schools when teaching algebra or universities when teaching biology. Based on that, we assume that adaptive learning is effective at least for these two categories of learners—school kids and university students. We want to go further and try adaptive learning with grown-ups—and see what happens.

Learning Management System (LMS) used to introduce adaptive learning needs to be able to:

quickly build remote courses out of small bits of theory (1-5 minutes) and separate practices;
create correlations between theory and practice, so that it is clear that a certain question from the test is related to a certain video/longread;
assign separate bits to individual users;
do follow-up and gather stats on these bits—whether a student was successful, their grade and number of attempts;
store user information (their 'profile') and update it depending on their success.

Requirements to instructional designers who will be building the adaptive learning-based program:

Understanding of the subject matter (meaning, an ID with specific knowledge of the subject, rather than an ID with general understanding of it);
Ability to divide the subject matter into separate independent bits to be studied for 1 to 5 minutes;
Ability to build correlations between these bits independently or with an expert—meaning, show users which pieces they need to learn first so that they get access to the rest of the material.

Theoretical framework

As a starting point, we chose knowledge space theory, which lies at the foundation of such distance learning tools as ALEKS and Knewton. For a detailed description of the theory, see our article (in Russian) or check out a recap below:

A domain of knowledge is modelled as a family of subsets, for instance, 7 subsets from a to f (in real life, there are 400 subsets). Note: although we use the term 'knowledge', each subset may be a purely practical one, such as an ability to construct an angle bisector or to fix a blank on a lathe, meaning that we include skills into the category of knowledge subsets.
Then, precedence relations are established between the knowledge items, i.e., to move to c one first needs to learn about a, in other words, a becomes a prerequisite for c, while d and e are the outcomes of c. This way, we visualise the knowledge space in a directed graph.
Before starting with the learning process, each student does a test to determine their level —a, d or f. When a user answers questions related to each subset, the system registers whether they are familiar with that area and offers them only unfamiliar concepts to learn. This way, we save students' time and keep them happy, because they do not have to return to pieces that they've seen 'a hundred times.'
During the learning process, students are offered unfamiliar pieces separately and after they successfully complete the test on each area the system automatically offers them new pieces in a designated sequence (see arrows), thus creating a personalised learning path for every user.

Method description

The method includes two stages—preparation and learning, with preparation being the key stage that is also the most resource-intensive.

Preparation

1. Choose a subject field—for instance, Java programming or project management. It is important that the field be a practical one, because we aim to teach users to actually do something, like set project goals or create cycles. Abstract notions and concepts wouldn't work here.

2. Optional stage (which we will include)—select a smaller area within the field or topic (for instance, naming critical success factors for a project). This step is a necessary one for smaller teams like ours, because modelling an entire subject field would require an effort of 100 specialists and would take about a year. Moreover, such an undertaking takes a lot of money, so we decided to start small and to take a closer look at the procedural standards as we go.

3. Outline the body of knowledge—a numbered list of all statements within the designated part of our subject field (there are roughly 20-30 statements, though we have not arrived on the exact number yet.)

4. Develop questions for every statement, such as 'is it possible to be familiar with this statement without knowing about that one?' or 'is it necessary to know this piece in order to learn the other one?' The questions will help to...

5. Form relations between the concepts. It would have been perfect if all the concepts were aligned like the letters in an alphabet, however, in real life there are many parallels, mutual dependencies between prerequisites and other items, mutual outcomes etc. A graph below visualises approximately 90 concepts:

In our program, we will have 2-3 times fewer statements, but it is obvious that the task is not an easy one. It will take 2-3 experts to take a look at the questions from the previous stage and answer 'yes/no', so that we assess the relationship between the statements, see which ones are interconnected and which ones are independent and after that draw the lines between them based on that understanding. Finally, the domain model is ready and we can start generating learning content.

6. Write practice tasks. Since we use practice as a starting point and select a practical subject field, these tasks precede theory and help remove the unnecessary theoretical concepts from the course. Each statement will include at least two tasks of both formative (aimed at practicing a skill in a safe environment without grades) and summative (graded assignment) types.

7. Write a detailed theoretical component for each statement, as if a student who has to learn about the area has no prior knowledge of it whatsoever.

8. Offer examples which illustrate how the theory is applied. This way, a statement will look like a combination of 'theory+example+formative assessment+summative assessment.'

9. Develop instructions for traversing between statements. So far, we've come up with the following:

User:

Completes first formative assessment (we have two of them, remember?) and launches the first summative assessment (out of two).
Fails summative assessment—receives second formative assessment and second summative assessment.
Fails second summative assessment once again—comes back to theory for revision.
Successfully completes any summative assessment—checks for relative prerequisite items and studies them before completing every one of them and receiving an outcome statement to solve. For instance, a student successfully completes c, then we check if they completed b and if they did, they are offered d or e. If they did not complete b, they need to solve it first to move on to the next item.

11. Establish rules of adaptive testing (when each new question depends on the candidate's response to the previous question). We already have the questions (in essence, it's a summative assessment based on the statements), but we need to establish rules, and to do that we need to consult with an expert to determine which concepts are key so that they are placed in the beginning. The complexity of a statement will increase or decrease depending on whether the user offers a correct response.

12. Select a system which will serve as a foundation for the program. It has to be an automatic system able to build relations between items, develop adaptive tests and gather stats. We are still looking for such a system and now we are choosing between SmartSparrow which offers a free plan for 5 members and a company with a customised system which will be able to implement our ideas.

Learning

1. Open the system for users and see how the rules work. If something goes awry—users drop out en masse, cannot traverse from one statement to another or are generally unhappy—we engage and modify the process as we go.

2. In the end, we assess the learning outcome—how users studied the material, how many correct and incorrect responses they gave and how many attempts they had.

Do's and don'ts

Advantages:

Saves time for students and makes them happier in the process;
Saves time for professors—they do not have to revisit the same topic over and over again and can concentrate on assisting their students instead.

Disadvantages:

Resource-intensive—developing an adaptive learning program for a subject area requires tens of thousands of person-hours;
Students' limited freedom—they have a pre-defined trajectory with few options to choose from.

Test design for the method

Test design to measure the effectiveness of the method

Prepare a set of content units for online learning (slides/videos, test questions). Organise learners into two groups:

Group A (experimental)—to study content units arranged into an adaptive track which adjusts to their knowledge state (with adaptive testing and hierarchy of statements);
Group B (control)—to study content units arranged in a linear track. All users receive content in the same pre-defined order as they would in an online course.

Preparation

Make sure the LMS meets all the requirements and that your team has the necessary experience (See Target Audience section.)
Recruit two groups of learners of minimum 10 people in each.
Introduce them to the rules of the learning program, but do not let them know that this is an experiment.
Optional for a blinded experiment: select at least two researchers, so that one prepares anonymised data and the other one makes calculations.

Conducting test and measuring efficiency