August 4th, 2009
Posted by Tihomir Bajic, Development, author of onebookaweek.com
The more you have riding on an idea, the more it hurts to be wrong. Human nature sometimes tricks us into perceiving disproved assumptions as failure, which can stop us from carrying on with the original idea. Successful people actually do the opposite, persisting and adapting even in the face of failure. Thomas Alva Edison once said:
Our greatest weakness lies in giving up. The most certain way to succeed is always to try just one more time.
Armed with such guidance from illustrious entrepreneurs, we set out to provide Rypple users with frequent, insightful, and actionable feedback.
Lukewarm Acceptance
We believed (and still do!) that asking concise questions to a select group of advisers who would then respond anonymously would be the ideal means of achieving that goal. We started with market research, ethnography, and observations of existing social behavior like Twitter, SMS, and IM, concluding that adviser’s responses should be limited to 140 characters (research shows that 160 is usually enough and we wanted to reserve a 20-character buffer for special characters).
We then implemented our first response box that prevented users from typing in more than 140 characters with a so-called ‘hard limit’, stopping them at 140 and not allowing the form to be submitted. Everything indicated that this was a smart decision that would drive high response rates, since users would find it obvious that a simple short response was required (and that they should dispense with pleasantries in order to be direct). With that design in place, we set out to conquer the world.
Lo and behold, we encountered lukewarm acceptance and some very vocal and disapproving users.
Lukewarm acceptance is more bewildering than outright rejection. — Martin Luther King, Jr
Back to the Drawing Board
This is the point at which failure might have tricked us into abandoning our cause. Mere mortals might have shrugged and walked away, falsely assuming that there was no solution. Fortunately for the growing population of Rypplers everywhere, we’re infected with startupitis, which carries with it a certain blunt stubborn refusal to accept failure (see our previous post about Rudyard Kipling: “if you can keep your head when all about you are losing theirs…”). We had expected that not everyone would accept these limits on their communication, but we were confident in our ethnographic research that had lead us to targeting Gen Y users with our alpha prototype, a cohort well familiar with the restrictions of SMS and used to conveying their thoughts in short, focused blocks. Still, some users truly struggled to provide a single response. What to do when all signs pointed to our assumptions being wrong?
Turn to the power of Agile and the ability to release early and fail often. It was obvious that releasing a working prototype and observing real usage was more rewarding than isolated academic research and analysis translated into software requirements. Our weekly iteration cycle let us focus on implementing one change at a time, measuring its impact, and then reflecting and making adjustments to our hypotheses. This scientific method introduced rigor that would serve us well in our battle for delivering a usable response box that would empower responders to give quick high quality feedback more often.
Modeling Cognitive Models
User observation showed us that people were used to brainstorming ideas and then pairing them down to get the message essence across. So our second iteration of the response box allowed responders to type in more than 140 characters but did not let them submit until they edited it to 140 characters in length at most. This was our first step to a softer limit, still restricting the length before submission but better modeling users’ creative process (raw creation mode followed by structured edit mode).
Overcoming the Blank Slate
This lead to people spending less time on writing a response and increased response rates of return responders. Progress! The next thing we observed, however, got us worried. Much like the dreaded blank sheet of paper, we realized that some users got stuck staring at the empty response box. We started experimenting with showing users great questions and responses, hoping to seed their creative process with inspiration. We showed them inline or in the sidebar or even as example text in the response box. We varied the help text based on tags used to describe the questions.
Expert Opinions
Our response rate increased as people overcame their response writer’s block, jumping by 5% over the previous condition. But now we noticed that the quality of questions and answers was dropping. We measure quality as the ratio of helpful to unhelpful feedback, as reported by Rypple askers. The amount of unhelpful feedback increased by 30% after the change. Grrr! Progress on one front was causing a regression on another.
Luckily, Rypple has attracted the attention of professional feedback coaches like Jamie Resker, Jennifer Stillings, and Cheryl Sylvester (to name only a small sample of our awesome community). Based on their advice, we added some help text to the form to provide guidance to advisers on giving better feedback, and we helped advisers by explaining what type of feedback was requested (whether it was a free form advice or a guided feedback response on what was done well and on what needed to be improved).
Sometimes Quantity is Quality
This helped us get back on track with feedback quality and kept the response rate high but we still had many vocal users telling us about how limiting 140 characters were. We then started experimenting with different lengths, sizing of response boxes, font size and text copy on the page. Allowing longer responses did not necessarily lead to higher response rates or to higher quality (which supported our original hypotheses about directness and brevity). A session with our friends and users at Mozilla helped lead us to the answer: the feeling of freedom from suddenly having more characters to use enabled users to provide feedback in shorter periods of time.
Our response length analysis actually showed that 200 characters were enough in the vast majority of cases. Only some types of questions warranted longer responses. Speaking to our users, deep diving into our data to look for trends, and iterating quickly allowed us to confidently increase response lengths up to 400 characters and quickly try several soft limits between 140 and 200 characters. We ultimately settled on a maximum length of 400 characters, but with the character counter starting at 200 and going to -200 before the form blocked submitting. We also added a series of short prompts that appear in increasingly darker shades of gray, providing feedback about how your response will be perceived by the asker (e.g.: “A concise response will be more helpful to Tihomir.”, “Your response is now longer than average.”, “70% of responses are more concise than this.”, etc.).
Empowering Advisers
This helped us with responses rates but we still had users complaining that some of the feedback did not make sense. Up until this point, we had been working under the assumption that it made sense for the asker to determine the type of feedback they wanted to receive. James Wu, our User Experience Guru, observed that face-to-face feedback doesn’t work that way: I ask you a question and then you decide how to structure your response. We ran this by our feedback coaches and partners who agreed, and went a step further by showing us the power of providing feedback using the coaching metaphor (e.g. what the asker needs to stop, what he/she needs to start, and what he/she needs to continue doing to be successful). We made a final change (so far!) by switching to giving advisers a choice of the type of feedback they wanted to give, choosing from ‘Freeform’ (a single field), ‘Like/Improve’ (two fields, 400 characters each), or ‘Start/Stop/Continue’ (three fields, 400 characters each).
Getting Better!
This brings us to how Rypple’s box for responding to feedback requests looks now. Switching to a soft landing for character limits and moving the decision about feedback types to the adviser has increased our overall response rate by over 15% and kept feedback quality steady. This is by no means the end of the road as we clearly see room for improvement and refinement. We’ll continue to use your invaluable insight and work together with you and the rest of the Rypple community on creating a useful and delightful feedback tool.
Tags: 140 characters, agile, cheryl silvester, customer outreach, decision support, ethnography, fail, failure, Feedback, feedback is fundamental, gen y, im, iteration, jamie resker, jeniffer stillings, martin luther king, peer review, personal review, professional development, quick polls, Rypple, sms, team review, thomas edison, twitter, usability, user observation, user testing
Recent Comments