For the last few months, I've been designing a seating chart generator for teachers called Shuffle Buddy. It generates randomized seating charts but allows teachers to specify rules about who should be seated where.

After months of work, I published my extension and put it to the test with teachers at the Knowles Teaching Fellowship spring conference.

My goal with user testing was to discover design usability issues. To determine the value of this kind of testing, I ran an experiment. Before testing with users, I wrote down a list of every usability issue I anticipated people would have. Then I compared that to the actual results.

The gist: Out of my 14 predictions, only 3 usability problems actually came true. I discovered 21 new usability problems that I did not anticipate (including 1 bug, 1 necessary feature, and 19 usability/design issues). In short? I had no idea how people would use my tool until I watched them try.

These results were so drastic that I am forever sold on the need for testing with real users. Results this extreme are not just a prediction skill issue. It is impossible to know how other people will interact with what you've built until you've put it in their hands.

The Data

For my experiment, I wrote down a comprehensive list of every potential usability issue I could anticipate before testing with real users. Then I wrote down a list of every issue I saw users experience during real testing.

While watching users try Shuffle Buddy for the first time, I was careful not to give any guidance about how to use it.

Every predicted issue and unanticipated issue is listed below.

Predicted Issues

The table below shows every usability issue I anticipated along with notes about whether or not it came true.

Prediction	Reality
Drafts & saved charts within a class. Users will have trouble managing versions. Requires people to understand a mental model that isn't made explicit/super clear.	Never got to test this; everybody just made one chart for one class
Classroom layout editor: Tags are confusing. How do you create them and assign them to desks? It would be nice to drag and drop a tag onto a desk or all the selected desks. You also have to connect the tags on the desk to the rules for the students and that connection is very indirect & unclear. The tags exist in two different places. Can the rules just directly assign students to particular desks? Do we need the concept of "tags"?	✅ Partially correct. People were confued about how to assign tags to desks, but once that was done the connection to the rules was straightforward
Bugs & glitches with copy/paste (in spreadsheet and room layout editor)	🚫 Nobody ran into glitches with copy & paste
Bothers me but I predict won't bother other people: Tricky to position desks at same sizes and nicely aligned. Can't rotate desks.	🚫 Nobody cared about this, but some people wanted other table shapes like hexagons.
Ugly but not confusing: The initial home screen which doesn't have a nicely designed empty-state. Not confusing because there's only one button, so you can't screw it up.	🚫 Nobody cared
Wondering: Will people know what I mean by the terminology "class"? Is it better to use "hour", "period", "class period", or something else? I predict that nobody will be confused, but they could be.	🚫 Literally nobody was confused for even a single second
People will not know what the checkmarks or circles with a line through it mean on the student names spreadsheet. People won't know that it means students are or are not placed in the room. It will only make sense if users have manual control over placing students. Part of me wants to be super opinionated and say that you aren't allowed to place students manually because you should set up rules instead, but that probably limits the tool to not work for use cases I haven't anticipated. Shuffle-with-rules-only is likely impractical and too strongly opinionated.	✅ Correct! The circle with a line through it was interpreted as a delete button, which it is not. I get the impression that nobody understood what this button does.
If you don't have enough desks, the generate button fails silently without any feedback. (It just doesn't do anything when you click it.)	🚫 Nobody ran into this because people set up their real classrooms which had enough desks for all their students. Many did not find the disabling desks feature, so this didn't come up.
Not clear how to delete students from the spreadsheet. I am curious how much of an issue this will be.	✅ Correct! This came up. We need a delete button in addition to the backspace option.
Empty rows in the spreadsheet are still treated as a student and "places" an empty desk. I bet there will be lots of room for confusion there.	🚫 Nobody explicitly pointed this out as an issue, but some had empty desks and I think it was an invisible problem that would eventually bite people later.
Will anybody rename a student (change the name in a particular row of the spreadsheet) and then be confused about the rules not matching their expectations? I predict that this will be a rare issue because I think most people will set up their names first and the rules later. I think it will be fine but it could be an issue.	🚫 Nobody ran into this
There are lots of easy-to-click delete buttons. The button to delete an entire class has a confirmation box, but deleting a seating chart does not. I wonder if people will accidentally delete things (like seating charts), for example when they intended to view a chart instead.	🚫 Nobody ran into this
The generate button doesn't gray out or give other feedback if you haven't entered your students yet.	🚫 Nobody had an issue with this
The Skyward import straight up doesn't work, but I think people won't even find that button.	🚫 Nobody found this button. I don't think anybody used Skyward.

Unanticipated Issues

The following is the list of issues I jotted down while watching people use the product. This list is unmodified except to redact the names.

Student Names Editor

Not obvious how to delete students (want a button)
- [Name] wants a delete row button in the student name entry
- [Name] expects the cross out circle to delete the row
- There seem to be people who are keyboard-based and people who are mouse-based, and it should be possible to do almost everything using just one or other other
It’s annoying that first has to be on the left and last has to be on the right. There should be an easy way to swap them because if you copy and paste from a report (like in PowerSchool), it might be backwards.
Need to flip order of columns so that I can paste data that is a list of last, first (not a spreadsheet, just text)
Doesn’t copy-paste correctly: Last in two parts comma first ([example redacted])
People were looking for an import button that allows uploading a CSV of student names, labeled “import” or “import roster”
[Name] didn’t realize he could tab to create a new student and was hitting the +Student button repeatedly.

Room Layout Editor

People want a save button on the edit layout dialog (and on everything, really)
People don’t realize they can select multiple desks at the same
It wasn’t clear that it was possible to shrink the desks, which meant that the square classroom shape didn’t work for a really long rectangular class.
If I shrink some of the desks and then add a new desk, the new desks are not the same size as the ones I already have
Glitch where desks can’t be moved–they just jump back to starting position
- In two cases, this happened after deleting all desks and then placing a group of 4
- There is not any message in the console
When editing desks, wants to ctrl+click to select multiple (only shift works currently)
Filled in bounding box makes it look like desks are selected when they actually aren’t (if they are in the box but not selected). Make the bounding box not filled in and make each individual selected desk more obvious.
“Import layout…” has been confusing to everyone (“what file type would I choose?”). If you keep this feature, it needs a different word. Maybe as straightforward as “copy layout from another class”.
[Name] wanted the group of four desks to stick together while dragging rather than just one of them moving at a time.
[Name] wanted hexagon tables that seat six.
[Name] (and many others) didn’t realize you could drag to select multiple desks. He was shift-clicking 1-by-1 instead.

Rules & Tags

Keep together option needed
- We definitely need an option where every person in the group is placed next to at least one other person from the group
  - One use case: Keep every girl with at least one other girl
  - Another: Keep every student with at least one friend who they like to work with
- We probably ALSO need an option to keep a complete group all together. This typically will mean a specific pair, so technically the previous keep together option would already support this
Possible feature creep? Add the ability to make a kid avoid a certain tag.
Nobody figures out how to disable certain desks. [Name] doesn’t want to have to choose, she just wants to have it prefer not to have stranded kids (keep everybody together by default)
- I wonder if I could keep the disabled option, but have smarter defaults if people don’t specify what they want
When creating a keep apart, wanting to type a list of numbers rather than one number at a time
- Don’t use dropdowns! Instead, have checkboxes. At least on the initial creation. That makes it easier to quickly check an entire group of people instead of having to re-scan the list from the top for each individual kid