Designing learning content at scale

A sneak peek into how we use data to design learning content at EntryLevel

When rebuilding our data analyst program at EntryLevel, I reviewed our own analytics and student feedback. To say it was varied would be an understatement. Some students loved the program, while others hated it. Some found it to be not beginner-friendly at all, while others said it was too basic. More than 3000 students have taken the course.

Although I knew the areas we needed to improve, the feedback was making me feel like I was going in circles. That's when I remembered a study I did on learning analytics in higher education back in 2016 for my research methodology class at university.

In the report, we discussed the field of Learning Analytics and how big data can be used to enhance the student learning experience. We highlighted the benefits of analytics for teachers, institutes, online course developers, and students, as well as the challenges and ethical issues that come with it. We also provided an overview of the tools and techniques used in this field.

But when I read the report again, I realized it felt super dry. While everyone knows the benefits of data and its analysis, what you do with it is what truly counts. For instance, data can be incredibly valuable in creating personalized learning experiences, providing early interventions for struggling students, designing course assessments and content, and ultimately improving student engagement and participation.

Customized learner experiences are still rare in online learning and MOOCs. Although colleges and schools may claim to provide personalized attention, I feel that's often just a marketing gimmick unless it's a small cohort.

At EntryLevel, customized learning is tricky because we have hundreds of students taking a course each month.

📊 Using data to create better courses

So, how do we use data to improve our courses? After every cohort, we look at these key metrics:

• On Mission Feedback (OMF) - feedback gathered at the end of each mission of the course (missions are sub-topics inside each course module)

• Module Churn rates - user drop-off rates after each module

• Qualitative feedback from completers and non-completers

Based on these metrics, we create a plan to revamp a course or a particular module in a way that:

• increases OMF

• reduces confusion and doubts

• adds more value to the student's learning experience

• helps them create a portfolio showcasing their learnings

For instance, our previous module on Tableau included just a small video on using Tableau and then jumped straight into a task. However, student feedback told us that it was confusing to use Tableau for the first time, and they didn't know why they should use it. Students also faced issues with uploading data and using the various features available.

So, we revamped the module on Tableau to include:

• An introduction to Tableau with four real-world visualization examples

• An explanation of different Tableau products

• Why it's better than Google Sheets or MS Excel

• A video on creating an account and uploading data

• A tutorial video on creating basic visualizations

• A list of extra resources if they're stuck

While I hope this improves the student learning experience, I'm aware that there will always be a few students who want more or less and are not happy with the offering. It's hard to please everyone.

What do we do in such a situation?

Designing courses at scale is a challenge, but we have a few tricks up our sleeve:

👥 Tailor Course Design Based on the Largest User Base

Our key audience is young grads getting started in a new field or people switching careers. So, basically, beginners. We serve a global market, but most of our students come from Nigeria and India. Keeping this in mind helps us design programs for complete beginners and provide more than necessary instructions.

Expert users or people familiar with using Excel might find some content too basic, but they're not our key users, although they may still find value in the Tableau modules. If we were building a data analyst program for Excel experts, we might move to more advanced concepts and use jargon without worrying that it would be too confusing for students.

🎮 Use Staged Rollouts of Content

Revamping an existing course can be a daunting task. Instead of creating a completely new program, we roll out new content in stages to different cohorts to measure feedback and see if students like the changes with minimal risk. By doing so, we can better understand what’s working and what needs to be improved.

For example, we might update task instructions for the March Cohort and then update the video in April (if needed) and based on the feedback, move with bigger changes.

😶 You Can’t Please Everybody

Designing online courses for thousands of students is a challenging task, and it's not possible to make everyone happy. While listening to user feedback is important, there will always be criticism, even if it's just from a few hundred users.

I remember when I was doing my bachelor's in Sydney, some students loved the program, some hated it, and I was somewhere in the middle, loving some parts of it and hating others.

This phenomenon isn't unique to courses; it happens with any product or service. Whether it's a restaurant or a laundry shop, you will always receive negative feedback because everyone is different. Even Apple, with its massive user base, has users who don't like its products, and the same goes for Samsung.

What's crucial to remember is to keep a check on your assumptions and be willing to listen to your users' needs. If a significant portion of your users wants something different, it's essential to adapt to their requirements.

If you’re curious, you can check out all programs on EntryLevel here. My favorites are Product Management and VC Analyst. And reply to this email if you want a generous discount code ;)

---

That’s it, folks. See you next time!  

Support: If you have a friend who is looking for learning inspiration and thoughtful articles, please forward or simply share this issue for some support.

If you have any feedback or comments, I’d love to hear from you, just reply to this email.