How Product Designers Observe the User Experience
As a university student, I studied computer science engineering because programming is a skill that can be used in any industry. Throughout college, I enjoyed learning about user interactions with software, and I was interested in understanding the reasoning behind software development decisions. As I explored how decisions should reflect user needs, I developed an interest in user experience, so I complimented my degree with some specialized courses.
My first contact with the OutSystems platform was during my thesis (programming a user-friendly iPad version of OutSystems Service Studio). I jumped at the rare chance to work in the area of user experience then, and now, here I am, working in user experience on the Product Design team at OutSystems engineering.
My main goal is to find out what users are thinking. When developing OutSystems, it’s essential that we gather data on how people are using our product, what they are thinking, and what they are feeling. There are several ways to do this, and today I bring you one of those research methods: observations.
What Are Observations?
As a research method, observations collect qualitative data from the user’s organic interactions with a product. This data reflects the use of our product in the real world and creates a record of actions that may have gone unnoticed. Analyze correctly and the user benefits, the product benefits, and future development will be focused on key improvements.
There are, however, a few drawbacks of performing observations with our users. But first, let’s look a little deeper into why we perform observations.
Why Make Observations?
The OutSystems platform was first developed 17 years ago and has been under constant improvement since. It’s very large and complex, but the user interface reduces complexity and allows customers to build large and small applications for web and for mobile much faster than hard coding. The Product Design team’s mission is to deliver a product users will fall in love with at first sight and keep on loving forever. The best way for us to understand what the users need (and love) is to observe how they use and experiment with the product. Every day, more and more people use our product.
We are growing fast and taking our technology everywhere. But everyone is different, so we need to get to know our users, their roles, behavior, needs, and expectations. Through observations, we can understand what the main improvements are, their priority, and have a better understanding of the differences between what we think and what the user thinks. What we learn from users is precious. The data helps us understand the mental model of the users, guiding us to design the product for a great user experience.
What We’ve Tried in the Past
When we started out, we lacked time to organize and plan observations but we took any opportunity to perform them. Earlier this year, a university hackathon using only the OutSystems platform was planned. So, we went to the hackathon to observe and tried using a low-tech method for registering what we observed: a notebook.
While great for note-taking, to-do lists, and doodling little pictures of hackers, it turned out notebooks aren't ideal for registering vast amounts of information to be later transcribed to a digital medium for data analysis, like Google Docs. So next we decided to just record directly into Google Docs, no more transcribing. But it was painfully slow to record and process the data. There had to be a better method.
Sometime later, we were in London conducting observations at a roadshow. Everything was happening very fast, it was hard to keep up with the pace, and we kept missing some of the events. My colleague decided to take out his phone and record what he was observing.
“User A1 doesn't understand how to use feature Y! User A2 couldn't find widget X.”
At the time, it seemed like a great idea. “We’ve found the solution!” we thought. But later, when we had to go over the audio and transcribe it, all our old problems resurfaced.
Frustrated, we tried using a tool called Reframer—we believed it was perfect for us. We could filter the observations by tags (one for each area of the product) to process the information. But again, we hit a wall. Processing the data was difficult because we had to keep entering issues that had already been observed.
What we needed was to be able to register the issue that we are observing with a one-click +1 on a list of issues. That way we could easily determine how many times each issue happened. With this tool, processing the issues was still time consuming because each observer entered their own issues in their own subjective way, so we still had a large amount of information to process. We were running out of options.
How We Collect Insights Now
At the time, it was clear that we needed a solution for gathering and processing the observation data. So we decided to build it ourselves using OutSystems, and the observation tool was born!
The tool has been loaded with all the scenarios that we’ve observed in the past. Using a search bar, we can easily find issues previously loaded into the tool. When we observe a behavior we want to register, all we have to do is locate the issue, and then click on it to store it as an issue found during the observation. The entry is associated with an observer and a user. Each issue can have additional information added to it, and we can also enter new issues.
With this tool, we halved our effort in gathering and processing the information from the observations. And now that we have the data collecting sorted, it’s time to look at the potential drawbacks of performing observations.
What Have We Learned?
We want to collect data from real-life scenarios, but usually, that wouldn't include someone observing you while you work, would it? So besides registering the observations, we must also try to control the various aspects that may lead users to be influenced by us. It really depends on the individual, but for some, having us lurking over their shoulder can feel very awkward. Some people’s minds go blank, others ask for help before even trying to do it themselves because they are embarrassed about potentially making a mistake. In these circumstances, it’s difficult for us to understand the genuine user interactions.
It’s up to us to put our users’ mind at ease. Therefore, we explain who we are and why we are doing this at the start. We also explain the importance of recording their mistakes and misunderstandings because this is how we can improve our product. It helps if we can be empathetic with our users. I like to tell them how the first time I used OutSystems I wasn't really sure what to do, that it takes practice and experience, and that the mistakes of today are the improvements of tomorrow.
Users Want to be Nice
At the beginning of an observation, we ask the user what they generally think about the OutSystems platform. Sometimes, they say, “It’s great, it’s very intuitive. It’s (almost) perfect.” We know that our product is great, but we also know that there’s always room for improvement. We understand that our users want to be nice, but we need an honest opinion so we can find ways to improve the experience over time. We need to know the objections, the doubts, the pains.
To achieve this, it’s very important that at the start of a session we put the users at ease; this is not a test! There are no winners or losers at the end. We really are interested in seeing how they use the platform, so it’s okay to have doubts and make mistakes. We can compile the doubts, mistakes, and misinterpretations and improve our product every day. They contribute to the greater good and a product with a fantastic user experience. Together, we are making developers’ lives easier.
Don’t Help Users, Understand Them
We’re only human, so this can feel uncomfortable for us, too. Even though we might be tempted, our role is not to help, it’s to simply observe. We must let the user experiment. Instinctively, we want to help them, and it’s hard to see users make mistakes we could easily fix, but that’s not why we’re there.
As observers, we have to accept this discomfort. The only thing we can do when a user is having difficulty figuring out a task is to ask, "What would you do next?" or "What you are trying to do?" or "Imagine that you were alone in your house doing this exercise, how do you overcome this difficulty?" We can’t offer guidance or hand out clues, and this can be frustrating and stressful. We feel the user’s pain and frustration. It’s essential that we never forget that this work is vital and the information we gather is precious. In the end, we only want to help our users have the best experience with OutSystems.
Let me tell you a story. I was conducting observations with a colleague at a roadshow in London. We were observing a user, and he was following an exercise that was scripted. To do the exercise, he had to start by opening a template inside OutSystems. He opened the template and kept it open in front of him.
Unexpectedly, the user sat there looking at the screen with his arms crossed. He was looking at a representation of a Splash screen that would be used in the application he was developing. After a while, I asked him if everything was okay. He said, “Yes, I am just waiting for the screen to load.”
The Splash screen in the context of the Service Studio is a static screen. After production, in a working app, a Splash screen is a screen that is displayed when the app is loading. The user had spotted the loading bar and was waiting for it to load. We felt an urge to laugh because this should have been intuitive, the way we represented the Splash screen had led the user to misunderstand what was happening, and it should have been obvious that the screen was only static, even for a first time user.
If he had been doing this at home, he wouldn't have us there to tell him this. He would be left stuck in this situation. This is why observations are so important. We want the users to have the best experience, we really do care. We had to explain our outburst, so we told him that it was a static screen. Fortunately, he thought this was very funny, and we all laughed together. At the time, it was amusing, but was it right for us to interject as observers? It isn't always easy to hide our emotions.
Wrapping Up
Observations lead us to new insights, and because of this, we have the tools to continually improve our product. For example, at an observation, we noticed that users were having issues with publishing their applications, and the users were attributing this issue to the platform's performance. But this was not the case; the platform was running smoothly, and the issue was poor network connectivity. At our office, we never encounter this issue, so we hadn't felt the user's pain. We realized that when connectivity is compromised, it should be visible to the user to ease their frustration and help them understand and fix their issue.
We are building a product to serve our users’ needs and desires. If we don't get to know them, then we are designing in the dark. It'd be as if we’re ordering a meal for a complete stranger. How would we know what to order? Getting to know the people that use our product is a vital ingredient in designing a better user experience and creating a product users love at first sight and keep on loving forever. What we learn fuels not only design decisions for the future but also informs what we could have done even better in the past.
There’s a lot to learn, and each day we take a step forward. The more we learn from the data we gather and process, the more we can put ourselves in the user’s shoes and anticipate what would make the user experience even better. With user observations, we are making OutSystems developers’ lives easier. And that is our mission. Having a positive impact on people’s work is huge, and we take our mission very seriously.
