During the Week 5 “explain the library website” test, one participant had a lot to say about Summon. He wasn’t happy about the change, although he did say that “it’s good, but it’s different and I’m an old man” (he was in his mid-20s at most). I wondered if he would prefer a newer style of search results page — the bento box (see this Musings about librarianship blog post for an overview with screen shots). I showed him a site with bento box results and asked him if he found this better than our Summon results. His reaction was immediate: “No, not in the slightest; this is more annoying… Maybe it’s just because it’s different, but I don’t like it.”
I found this interesting because a number of libraries have moved to bento box results (e.g. UCalgary, Simon Fraser, Duke, Stanford, Princeton, NCSU) and I find the concept quite compelling. Was this guy, who admittedly did not like change, an anomaly?
So for Week 6, not only did we look at how students were using our Summon search, but we asked them to do the same search on one or two of the bento box sites to see if bento worked better. Chocolate? Check. Off to the lobby!
Overall, students had very little difficulty with searching Summon. Although in our “explain the library website” tests, students mentioned Boolean a lot, only one of our four participants used a Boolean search in Summon. Two used phrases of four or more words. All of them saw the filtering options on the left side and appreciated that they could narrow their searches in this way.
A few problems we noted:
- No clear ability to limit to eBooks
- Confusing to have Hold/Reserve button appear for Reserves items
- SFX link resolver interface very problematic (moving to 360Link should help)
Bento Box Results
Of the five students who saw the bento box results (4 this week, 1 the previous week), three disliked the whole concept instantly. Their reaction was immediate and visceral: “Oh, oh, this is confusing, I don’t like this.” “Uh, um, I don’t really understand what it’s telling me.” “Ohhhh… Not a big fan.” I edited the audio of these reactions together and it’s a symphony of disgust.
Those who quite liked the initial results screen were then unimpressed when they clicked “more results” or the Articles heading and were brought to a discovery-style screen. They both said they would have preferred going to the discovery screen directly, one continuing “because it already has all the filters and everything here.” They did not see any value in separating out Articles from Books when Summon allowed them to do that with the facets on the left. And they weren’t interested in any of the other bento options beyond Articles and Books.
I was surprised that none of the students liked the bento box approach. I still think there are use cases where bento would be an improvement, but I don’t think it balances the negative experience for the majority of our users who really do just want articles and/or books.
One thing that makes me really happy about all of this was that we saved a lot of time and resources by testing the implementation of the concept at other libraries instead of starting to develop it ourselves. It’s certainly possible that our five random students were anomalies and that bento would be a big improvement for our users. But five out of five negative reactions is enough to let us decide to put our focus on other projects. We may well revisit bento again, but only after I hear cries of delight drown out the cries of dismay.
2 thoughts on “Weekly user tests: Searching”
Interesting results. I believe there was also one study done in the UK that found undergraduates didn’t like Bento,
Of the ones you listed Duke, NCSU, Princeton etc they are pretty research intensive, and to a certain extent bento works well with users who have well developed mental models and know what they want.
As a added bonus bento helps out with known item search of catalogue items in Summon due to the seperate catalogue box which helps pacify the faculty from the none STEM areas who just hate that the results get flooded sometimes by articles.
For example try Making Democracy Work Putnam (a real life search I saw via analytics). , The actual book is buried beyond the top 10 even if you exclude book reviews. Faculty here just hate it when they enter title (partial) + Author and it’s buried. Sure you could just filter but it’s annoying to them.
Over here we developed a bento style result page by altering the libguides v2 search template to pipe in Summon results.
See http://nus.beta.libguides.com/srch.php?q=systematic+review . So far we plan to push it out to the libguides search only where it is clearly an improvement from just showing libguides.results.
Off hand, I am not sure if we will push bento style from our main portal page. That’s a bit more risky, though my institution profile counts as research intensive.