Your rollout schedule will probably determine how long you'll be able to conduct the usage portion of your beta test. We recommend giving it at least a week to get as many people through the site as possible.

Once you've completed the user testing, it's time to get some closure, as they say. Take all the e-mail addresses of the users who agreed to participate in the test and send them another solicitation. This follow-up survey will be used to get their overall impressions of your site's performance and most importantly, their satisfaction with the experience.

Answer the Questions

In your followup research, you should look for the answers to two questions:

1. Did they use it?

As we mentioned earlier, one of the most powerful pieces of information gleaned from a beta test is an understanding of the types of customers who became regular, moderate or light users of the site. Because this is the best real-world indicator of the level of value testers found in the product, it deserves special attention.

You can estimate usage levels simply by asking respondents about it in the follow-up survey. Beware that users have a tendency to overstate things like this, so be sure to discount their responses by some degree (20-30 percent is about right).

Evaluating the profile of testers who became regular, moderate and light users should reveal strategically relevant groups and, hopefully, match pre-existing segmentation hypotheses you had for this product. Your primary target segment should become the heaviest users, the secondary segment should become moderate users, and so forth. If the distribution of usage does not correspond to your initial segmentation hypothesis, it should be revisited to determine where the discrepancies lie. Perhaps something was lost in the translation from an idea into a real working product. Or perhaps this group of customers has other characteristics that make them more difficult to satisfy than originally anticipated. Whatever the reasons, you need to decide whether to pursue the original target or to modify your segmentation and marketing schemes to reach the consumers who became heavy users of the current product.

In addition to those who used the product to some degree after the initial introduction, there will be participants who failed to discover enough value in the site to return again. Evaluating the reasons for the attrition so that you can take steps to combat it is another major aspect of the follow-up survey.

2. Did they like it?

To supplement usage measures, we recommend collecting performance ratings of individual sections or features. This provides another dimension for understanding the role that each feature plays in the overall value of the site.

You can obtain performance and satisfaction feedback in a top-line or granular way. We'd suggest going for as much detail as possible without over-taxing your testers with an obscenely long survey. Ten to 15 minutes should be the limit. Since every response is critical, you don't want to risk losing someone's feedback because they were burned out by a difficult questionnaire. Because you collected most of the background information on the respondent prior to beginning the beta test, you can omit these questions (presuming you have been using a unique ID and can pass that through to the new survey). Stick to the basics:

• usage level of particular features/areas of interest.
• overall satisfaction with experience.
• satisfaction with features and areas of critical site performance (e.g. ease of navigation, search, etc.).

One approach we've found to gather great data with a minimum of respondent fatigue is to phrase the performance/satisfaction questions in a competitive context. For example:

How would you rate XYZ.com on the following criteria?

• Providing detailed online technical support.
• Ease of finding specific product information.
• Speed of download/access times.

Then ask respondents to evaluate each criteria by selecting the relative description that most reflects their opinions:

• One of the best of its kind.
• Much better than others of its kind.
• A little better than others of its kind.
• About the same as others of its kind.
• A little worse than others of its kind.
• Much worse than others of its kind.
• One of the worst of its kind.

Go for Consistency

Is the feedback you got from testers during the usage period consistent throughout the entire group?

The follow-up study is also the place to test the significance of the qualitative feedback received during the beta testing period. Perhaps a number of participants in the beta group clicked on the feedback button to inform you that the relevance of the site search feature was not up to par, or commented that the site should include more independent product reviews. The degree to which these opinions exist in the overall beta group can, and should, be quantified by including specific questions addressing these issues in the follow-up study. Unless you intend to devote your life to chasing down every comment made by a beta-tester, this is your opportunity to separate the issues shared by most users from those brought up by a fringe element.

Last-minute Tweaks

The follow-up study can also be leveraged to address any last-minute internal development ideas that arise during the beta testing period. Frequently, the observed performance of the site during the beta period prompts new site functionality or design proposals. The customer's need for and the level of appeal of these ideas should (as always) be tested before implementation.