Usability Testing in Online Course Design

by Justin Marquis Ph.D.

Every instructor who has designed his or her own course and taught it has actually conducted two rounds of informal usability tests. For the first round, you were the user, and in the second, your students were.  What is usability testing? It is simply a process of observing how users interact with something you have designed in order to evaluate its effectiveness and implement changes for the next iteration of the product. But there are drawbacks to the kind of informal testing you have done during your career as a practitioner of instructional design. First, you may not have been paying that much attention to the user experience, particularly in an online setting, as your audience is not present in a location where you can actually watch them. Secondly, you probably did not plan any specific usability tasks designed to test elements such as navigation and ease of finding information. Finally, if you are like me, it is unlikely that you took notes on the user experience during the actual course. Maybe you jotted an occasional note, or have some scattered emails from students pointing out issues, but it is unlikely that you have any sort of useful data that can be easily accessed and evaluated.

Traditional Usability Testing
Usability testing is usually a structured process involving multiple rounds of prototyping materials, writing scripts for the sessions, carefully selecting activities for users to complete, observations, interviews, and an evaluation of the results. For example, in graduate school I spent countless hours building (and testing) a usability testing station which used multiple computers, microphones, and video cameras; to simultaneously record the computer screen, the user’s face and hands, the tester’s questions, and the participant’s responses. All of these elements could be integrated into a single video stream which was saved and analyzed, as well as shown to the user during the post-test debriefing interview. This entire setup was housed in a gigantic wheeled cabinet so my colleagues and I could transport it to the users. So go ahead and build one, and let me know when you’re finished and we’ll continue…

This process is obviously impractical for the average course designer even though usability testing is considered one of the essential components of good website design. So much so, in fact, that we have an entire U.S. government office dedicated to it, The site provides resources on planning, user-centered design, developing prototypes, conducting usability tests, writing reports based on the tests, and measuring trends revealed by the testing. In addition, the site provides information on "usability basics," and hosts a collection of templates, forms, and examples for each stage of the usability testing process, all of which is freely available.

Breaking with Tradition
If a full-blown usability test is beyond your knowledge, ability, or patience, what can you, as an online course developer, do to ensure that the material you are designing for your students is well-conceived of and will accomplish your learning objectives? For starters, you can evaluate your materials as you produce them. From your profession as a teacher alone, you should instinctively know what is important. Rely on that experience and consciously think about what worked during previous times you have taken or taught the class. Make a list of things that worked and didn’t work, and incorporate or eliminate those elements from the course as you create it.

Realistically though, when you design or write something, you are generally too close to it to effectively evaluate it. You will need to get at-least one more pair of eyes to examine your course materials before making them available to your students. Here is what you, and anyone helping you, should be looking for:

  • Clearly stated objectives – Are the learning objectives easily accessible to the user and are they clear and easily understood.
  • Laid-out course plan – The class schedule should be a vehicle for navigating the class. Are hyperlinks used to direct students to resources, assignments, and necessary tools? The class schedule provides a centrally-located resource from which students can access all other course information and materials. Think of this as the homepage (which it literally could be) where students will go to find out what comes next in the class.
  • Intuitive navigation – Make sure that your organizational strategy is sound and intuitive and that you are providing your users with the necessary information to help them find what they are looking for.
  • Everything works – There is nothing more frustrating for students than being directed to a link to some critical material that doesn’t work. Web addresses change occasionally and resources have a tendency to vanish without notice. Just because it worked last term, doesn’t mean that it will work this term.

How to Conduct an Informal Usability Test of Your Online Course
First, create a series of “task cards” on note cards, putting one task on each card. Present these to your subjects at the start of the test and have them attempt to complete each one in turn. This means that you will need one or more test subjects who are somewhat representative of your intended audience. At the very least, they should have a similar level of subject knowledge and technical proficiency. Once you have secured your guinea pig(s) you should take them through the following process:

  • Locate the important components of the course – Have them search for and identify the course objectives, class schedule, syllabus, relevant discussion areas, video chat tools, the grade book, grading rubrics, and any other pertinent information that students will need to access on a regular basis to be successful in the class.
  • Verify the navigation strategy – Part of this will come as they work to find the resources mentioned above. But some of it will be based on your observations of how many false clicks they use to locate a specific resource. This may be largely irrelevant in a learning management system that lacks customizability, but you can always consider rewording the instructions in your course syllabus to more accurately direct learners where to find things.
  • Test links – If you can secure multiple volunteers, you can provide each with a different set of links to test during their session. Ultimately, between the testers and yourself, you want to verify every link before the class goes live.
  • Complete a task – Have your test subjects complete one or more authentic tasks that real class participants would have to do. This can include posting or responding to a discussion thread, accessing and using an online tool, watching an embedded video, or some other process that will be important to the class.
  • Use multiple Web browsers and operating systems – At the minimum, you should have your volunteers try Firefox and Internet Explorer’s most current versions on both the PC and Mac platforms. If you only have one tester, you can ask them to switch browsers part way through the session, but you do want to verify that functions like discussion boards, digital drop boxes and videos work on a variety of platforms.

Tips for Running the Usability Test
One of the biggest mistakes that novice usability testers make is not recording the information from the test sessions. This is a fatal flaw, as you are unlikely to remember even a fraction of the specific data that you uncover during the tests. Fortunately, there is an easy solution: take notes. You should conduct the test while sitting with the learner and make notes from both your observations and comments they make during the session. A good way to facilitate this commentary is to encourage (and remind) your testers to "think out loud" as they work through the tasks you have assigned them.

A second important tip is to set a specific time for the test, agree on that amount of time with the volunteer and don’t go over it. They are doing you a service and will be more likely to do it again in the future if you are accurate about what their commitment will be. A final and related point for conducting a usability test is that you cannot take the feedback or results personally. This is a process of critical feedback which you should embrace as a way of making your course better. Any problems that the test subject finds are not their fault. They are yours, either through poor initial design or a failure to clearly articulate the tasks to be performed. Be thankful that they donated their time, and revise your materials accordingly.

For the innovative online course developer, you can conduct this type of informal usability test entirely online using a video conferencing tool such as Skype, a digital whiteboard, or screen casting tool, and emailed directions. All you need to do is observe through video chat while your test volunteers try to complete the tasks you chose for them. This could even be good practice for integrating online video chat into your course design.


Image: renjith krishnan /