Doris Downs
Doris Downs / Development / March 18th, 2016

MixFrame Journal: A Developer’s Adventures in User Testing

As a developer, I am familiar with some user experience concepts, and I’ve long since learned to appreciate the great work our UX team does. However, our internal work on MixFrame offered my first experience with user testing.

For MixFrame, we wanted to be intentional with everything we did, from analytics to feature development – user testing and research was no exception. While our UX strategist Michelle interviewed developers, we also wanted to get our alpha product directly in the hands of our core users: front-end developers. After watching our product owner and scrum master walk other 352 developers through the setup, installation and use of MixFrame, I was ready to lead a first test myself.

mixframe-chris

The Reality vs. The Ideal

Developers often have an idea of how a product will be utilized, and as part of the product team we often cultivate a collective understanding of how our product will be implemented and used. Since I’m also in the core audience for the product we’re building, I felt confident in my assumptions about MixFrame.

In user testing, I discovered this understanding did not always match up with what users actually did when first exposed to the product. For example, MixFrame requires the user to add data attributes to the components of their style guide. It turns out that what our team considered components was sometimes very different than how our users viewed components. So, while we may have intended the user to place attributes on individual media elements, the user may have placed an attribute on the container.

We saw vastly different ways of structuring mockups and creating components that were unintended, but very informative. So while the user case was clear to us, did we need to improve documentation to align user assumptions more closely to ours or could we just adjust the product to make interactions more intuitive?

Fresh Ideas and Features

Building a new product requires a constant evaluation of our product roadmap, and it’s amazing how user testing impacted our outlook. New ideas presented themselves, and old ideas suddenly seemed much more important when users wanted and needed them.

Some were small features, such as adding instructions to a blank screen if the user set up MixFrame, but didn’t define any components. Some were epics that would improve on current features, and others were new epics in and of themselves. Some were simply ideas for improving our documentation and usability.

mixframe-drew

Renewed Excitement

While the obvious outcome of user testing may be new feature ideas and guidance, there was one unexpected outcome: these user tests were tremendously exciting. I’m always deeply invested in the success and usability of the products we build, but MixFrame is different.

Seeing actual users interact with the product and getting great ideas from their reactions was invigorating and encouraging. As both a developer and a stakeholder in the product, I can see the positive effect new features and updates will have on users.

User testing can be a great way to increase enthusiasm and buy-in on a project, particularly if the developers have found themselves in the so-called trough of sorrow (that period of malaise that often follows the initial excitement until market traction is found).

Your (Mostly) How-To Instructions

As I learned, the basics of user testing are pretty straightforward. Figuring out what to do with that data can be difficult.

We learned quickly that instructions and documentation were critical to user success; we also learned that some people really don’t like reading the instructions. Like me, many developers prefer to just dive in and see how things work. Additionally, many users had pre-set notions of how the tool should work, likely based on interactions with similar digital prototyping software.

mixframe-qaIt would be easy to ignore some of the fringe cases, and merely assume that users would eventually conform to our standards – but how many users would we isolate or drive away by forcing an interaction the way we thought it should happen?

This balancing act suggested the question: how many perspectives can and should we accommodate? Conventional UX wisdom says testing with 5 users is enough to reveal 85% of the potential problems with your product. Anything past that can muddy the waters and sidetrack the team.

While our tests did provide valuable insight, we had to focus on the most achievable insights first. To do so, I recommended incorporating as many ideas as we were reasonably able to in order to reduce user friction and increase the potential user base. However, once we reach the limit of what we can reasonably do, we don’t need to necessarily be worried. To some degree, as long as we build the product to a level that it’s deemed valuable, people would adjust to the product while the product also adjusts to them.

Ultimately, user testing is one of the most interesting and illuminating experiences I’ve had as a developer, and I highly recommend it. User testing isn’t just something that developers should react to – it’s an opportunity for developers to directly challenge and overcome their own assumptions, which can be invaluable to the success of any product.

  • Paul Traylor

    Couldn’t agree more Doris 🙂 Seeing users react with products we develop is a mind-bending experience at times. I almost always come away humbled and with renewed vigor to right the wrongs. The last thing any good developer wants their solutions to do is bring unnecessary frustration into people’s lives.