Lesson video

Hello, my name is Mrs. Holborow, and welcome to Computing.

I'm so pleased you've decided to join me for the lesson Today.

In today's lesson, we'll be evaluating the accuracy of a model and creating a model card to document a machine learning model.

Welcome to today's lesson from the unit Machine Learning Using the micro:bit.

This lesson is called Documenting and Evaluating a Machine Learning Model.

And by the end of today's lesson, you'll be able to create a model card to document a machine learning model.

Shall we make a start? We will be exploring these keywords in today's lesson.

Accuracy.

Accuracy, how many times a model makes a correct prediction.

Accountability.

Accountability, taking responsibility for how a model behaves and being honest about its performance, limitations and potential impact on others.

Model card.

Model card, a short report on how a machine learning model works, how well the model performs, and what people should know before using the model.

Look out for these keywords in today's lesson.

Today's lesson is broken down into two parts.

We'll start by evaluating the accuracy of a model.

We'll then create a model card.

Let's start by evaluating the accuracy of a model.

A confusion matrix is a way of visually showing the accuracy of a model.

Here's an example.

A matrix for a model that classifies movements as jump, spin or wave might look like this after 10 tests of each movement.

So we have the actual movement along the top, so we have jump, spin and wave, and then we have the model predicted down the side.

Of the actual 10 jumps, the model correctly predicted jump eight times.

Two actual jumps were misclassified as a spin.

No jumps were misclassified as a wave.

The model's accuracy can be calculated by the correct predictions divided by the total predictions.

So in this example, the correct predictions were eight for jump, seven for spin, and eight for wave.

Add those together and we get 23 correct predictions.

The total number of predictions was 30, so the model's accuracy is 23 divided by 30, which gives us 0.

7667, so the model is about 76.

67% accurate.

Out of 30 test samples, the model correctly predicted 23.

Accuracy also helps us take responsibility for our work.

If a model is being used in an app like a fitness trainer, we need to know how often are mistakes made, who could be affected if mistakes are made, and are the outputs fair to all users? Keeping track of accuracy and being open about mistakes is part of being accountable when designing technology that other people use.

True or false, to calculate the model accuracy, the total number of predictions needs to be multiplied by the number of correct predictions? Pause the video here whilst you have a think.

Did you select false? Well done.

You need to divide the number of correct predictions by the number of total predictions.

So the formula is model accuracy is equal to correct predictions divided by total predictions.

Calculate the accuracy of the model that produced this confusion matrix.

Pause the video whilst you complete your calculations.

How did you get on? This model is 90% accurate.

Out of the 30 test examples, the model correctly predicted 27.

Okay, we're moving on to our first set of tasks for today's lesson, and you're doing a fantastic job so far, so well done.

Using a model you trained in CreateAI, complete a confusion matrix for your model, then calculate the accuracy of your model.

As a tip, it is a good idea to have at least 10 test examples for each class.

So if you don't have those, you may need to record them.

For part two, use your confusion matrix to evaluate how accurate your model is.

Are there any classes that are misclassified more than others? Pause the video whilst you complete the tasks.

How did you get on? You were asked to use your confusion matrix to evaluate how accurate your model is.

You were also asked whether there were any classes that were misclassified more others.

Here's a sample answer.

"The confusion matrix stated that my model was 65% accurate.

The jump class was often confused with the spin class, so I may have to collect more training data for clearer motion differences." What was the percentage accuracy of your model? Were there any classes that were misclassified more than others? Okay, we're now moving on to the second part of today's lesson, where we're going to create a model card.

A model card is like an instruction manual for a machine learning model.

A model card could include the usage or purpose of the model, the training data that was used, the performance of the model, so maybe it's accuracy that we've just looked at, any limitations of the model, and any ethical considerations.

Why do AI models need documentation? Maybe pause the video whilst you have a think.

Ah, Izzy's got a response, "It helps people understand what the model does, the accuracy of the output, and any failures or mistakes the model could make." That's a brilliant response.

Well done, Izzy.

Time to check your understanding.

Which one of these is a reason for creating a model card? Is it A, to make sure that it can only be used by experts, B, to help the model make more accurate predictions, or C, to be transparent about the use and limitations of a model? Pause the video whilst you have a think.

Did you select C? Well done.

A model card is created to be transparent about the use and limitations of a model.

In this part of the lesson, you will create a simplified model card for a model you have created.

You will be given a template to use.

This is an example of the template.

So it's split down into a number of categories.

Firstly, the purpose.

What does your model do? Who are the users? The training data, what motions did you record? And did you include a null class? The performance, how accurate is your model? What worked well and what didn't? Limitations, when might your model make mistakes? And then lastly, ethical considerations.

How could your model be unfair? Could someone misuse your model? True or false, a model card can include accuracy scores as part of the performance section? Pause the video whilst you have a think.

Did you select true? Well done.

A confusion matrix can be used to calculate the accuracy score, and then this can be included in your model card.

Okay, we're moving on to our final set of tasks for today's lesson.

Use the template provided to complete the model card for your model.

You could visualise the results, so certainty scores, confusion matrix, et cetera.

You could also add tips for using your model.

For example, try to use a steady hand.

Pause the video whilst you complete your model card.

How did you get on? Did you manage to create your model card? Your model card will obviously look different from mine, but let's have a look at a sample answer to together.

So for the purpose, the model can tell the difference between the three movements using motion data from the micro:bit.

The model's output is used in a motion-controlled game where players make a character jump by using their own movement rather than the buttons.

Training data: I used lots of examples of each movement.

Each one was done 30 times by different people in my class.

I included a null class so the model doesn't get confused if there is no movement.

Performance: the model is about 67% accurate.

Predictions were accurate for spin, but the model mixes up jump and wave.

This might be because some people raised their hand to wave.

Adding the null class helped and reduced random guesses when there was no clear action.

And here we can see the confusion matrix for the model.

Limitations: the model might make inaccurate predictions if someone's movements are too small, too fast, or not like the examples used for training.

The model does not perform well if the user holds the micro:bit at a different angle to how it was held when the training data was gathered.

The accuracy is not worse for left-handed players who move differently.

Ethical considerations: the model might not work equally well for everyone.

If someone moves differently because of a disability, their actions may not be accurately predicted.

That wouldn't be fair, especially in a game.

Did your model card look similar to this? Remember, if you'd like to, you can always pause the video here and go back and add some extra detail to your model card.

Okay, we've come to the end of today's lesson, and you've done a fantastic job, so well done.

Let's summarise what we've learned together.

It's important to evaluate a machine learning model's accuracy.

A confusion matrix can be used to visualise this.

Reporting a model's accuracy improves accountability, so we can be more open about the mistakes and issues.

A model card provides more information, improving accountability, and demonstrates reasonable practise.

I hope you've enjoyed today's lesson, and I hope you'll join me again soon.

Bye.