Check out Robin Chase’s (Zipcar) TED talk about how mesh networks, social networking, and automobiles can affect climate change, eliminate barriers to technology, and open whole new economies. Fascinating talk.
In accordance with the 8 one-on-one sessions we performed in late January on Search Beta v1, we decided to give UserTesting.com a whirl, mostly because of its affordability. Generally, my expectations of the service were blown away, and I would definitely use it again.
We went with 5 testers for $95, which is the max amount of testers they permit at this time.
I ask the testers to:
Use the search to find a home that you might be interested in. Search a couple of areas if you don’t find one initially. When you find one, go ahead and inquiry about the property, but do now click the final “Submit” buttons. What was frustrating? What was easy? What features did you like or dislike? What features would you want to see added?
I also asked for:
Gender: Doesn’t Matter
Age: 35-55
Yearly Household Income: $40,000 to $100,000
Expertise with computers and the Web: Beginner/Intermediate
Other Demographics: home owners
Within a couple of days, I had all five of my replies, and it took me almost an entire morning to review them and distill the useful bits.
You receive both a video and a written review. The video offered a much richer insight into the user experience. Testers talk aloud, and you’re able to view their actual interactions with the software. Clicks are shown as a red circle. Creating this rich experience is, in and of itself, an accomplishment for UserTesting.com. This is about as good as it gets (with the exception of eye-tracking, which would bankrupt most companies).
The written evaluations are just bulleted listings, with answers to some canned questions. Most of my testers were good about matching their written responses with their comments from the videos.
My only real critique is that some of the testers simply described how they were navigating the site. For example, “I’m clicking on this listing.” or “Now I’m going to Foreclosures. This is the foreclosure page. Seems pretty straightforward. Now…” One tester didn’t even stay on task.
Comments such as these aren’t helpful. I need to understand what the tester is confused by, what they’re working out in their skulls. “Does this New thing mean ‘New Construction’ or ‘New Listing?’”
The website gives you the option to rate the testers, and I rated them according to how much constructive insight they provided. One was outstanding, and a couple of his suggestions (or confusions) will make it into v2. One was horrible. The others were average.
Like I said, I would definitely use UserTesting.com again, but I don’t think it replaces other forms of research (especially one-on-one sessions). It’s a terrific tool to augment and validate other research methods.