After so many of you attended the recent webcast we sponsored on “Findability: Designing the Search Experience” with Tyler Tate of Twigkit, there was a long list of questions that didn’t get addressed during the course of the presentation. As there was a pretty good list of them, we thought you might find it useful to get some more insight. So we’ve invited Tyler to put together some answers to your inquiries.

Q: Are there quantitative methods to judge the effectiveness of your search experience, like by analyzing logs of what people did — or is it really just about talking to people and watching what they do interactively?

TT: Search logs are really useful for judging and improving relevancy, but they only provide insight into individual queries. In reality, however, users may enter several queries or filter their results numerous times before finding what they’re after. What would be useful is if you were able to track query reformulation data that revealed the path a user followed during a given session. This could give you insight into how people actually use the UI over a longer period of time. That said, usability testing with real users is both easier to conduct and will more quickly surface the biggest flaws in your system, so I would always start there.

Q: Is there an optimal number of search results or optimal number of facets that can be readily apprehended by most users?

TT: I haven’t heard any universally applicable figures. I think it goes back to knowing your audience. Experts will desire more complexity than amateurs. And more specific tasks will allow for options that broad tasks. But as a benchmark, eBay and Amazon often provide 20-35 filters spread across around 10 different facets.

Q: Is there conventional wisdom about the fraction of your design or budget you need to devote to user experience design for your search system, say, relative to your overall development budget? Like for QA, there’s conventional wisdom about 1 QA for every 2 developers…How do you know if you are investing at the right level for your search experience design?

TT: Search UI frameworks (such as TwigKit, where I work) can reduce the cost involved in building the interface. Traditionally, however, about 30% of your time and budget would need to be devoted to the user interface in order to deliver a top-notch result.

Q: With large numbers of matches per filter or facet, how can you prevent a user from picking what’s “popular” vs. what’s relevant, — aka, the rabbit hole problem?

TT: This question is getting at the “needle in a haystack” problem. How can you find that one anomaly in a sea of results? The best solution that I’ve found is to offer an “exclude” option for each filter. That way, users can exclude the popular chunks of results and narrow down to the rare ones. It adds a bit of complexity to the interface, but in some cases it’s worth it.

Q: Can you comment on the tradeoffs between performance and findability? i.e., at what point is letting the search system work harder for better relevance or presentation going to disappoint the user’s need for speed?

A: Speed is crucial. If the interface is sluggish, users are likely to get frustrated or even stop using it. An ideal page load would occur in under 2 seconds, which seems to be the threshold of uninterrupted attention (See Nah at http://sigs.aisnet.org/sighci/bit04/BIT_Nah.pdf , Nielsen at http://www.useit.com/papers/responsetime.html). Factoring in download times, that means your server should definitely deliver a sub-second response.

Q: Can you offer comments on a test plan for your interface, such as for zero results … and so on.

TT: It depends of course on what you’ve chosen to include in your interface, but we typically test for the following scenarios:  * No query – before the user has begun their search  * No results  * No results but when a spelling suggestion is available  * Recall – is the engine returning relevant results?  * Results for the user’s query  * Results for the user’s query when one or more filters have been applied  * Results when filters have been applied but the query has been removed  * Results when spelling suggestions are available  * Results that contain empty fields  * Results that contain really long fields

Q: What type of security considerations do you recommend with total history and search ahead and facet counts

TT: A user obviously shouldn’t be allowed to see a suggestion or filter if they’re not authorized to view the resulting document, as the knowledge that the term even exists is itself a security violation. I’m not an expert on security, but how we’ve often solved this at TwigKit is by storing access details in a field on each document. We then append a hidden filter to the query that indicates the user’s security clearance, thus filtering out the documents he’s not supposed to see.

Q: Can you recommend an approach to presenting or grouping lots of results from a federated search collection?

TT: Federated search is really difficult to pull off and I don’t know of any silver bullets. I’ve seen it presented in a single merged result list (Kayak), segregated into tabs, or shown as a series of groups listed down the page (iTunes).

Q: How would you characterize a difference between searches done for an online store verses a public library catalog or museum

TT: I think that the tasks themselves (i.e. finding a book) are almost identical. Historically, library card catalogs have been more complex than consumer search interfaces. This is probably because in the early days, librarians were the primary users of such systems and were trained in entering advanced queries. Of late, library card catalogs do seem to be incorporating the simplicity that users have come to expect from web search engines, much to the benefit of the typical user.