Weekly user tests: Finding subject guides

This week we did a guerrilla-style test to see how (or if) people find our subject guides, particularly if they are not in our main listing. We asked “Pretend that someone has told you there is a really great subject guide on the library website about [subject]. What would you do to find it?” We cycled through three different subjects not listed on our main subject guide page: Canadian History, Ottawa, and Homelessness.

Some Context

Our subject guides use a template created in-house (not LibGuides) and we use Drupal Views and Taxonomy to create our lists. The main subject guide page has an A-Z list, an autocomplete search box, a list of broad subjects (e.g. Arts and Social Sciences) and a list of narrower subjects (e.g. Sociology). The list of every subject guide is on another page. Subject specialists were not sure if users would find guides that didn’t correspond to the narrower subjects (e.g. Sociology of Sport).

Results

The 21 students we saw did all kinds of things to find subject guides. We purposely used the same vocabulary as what is on the site because it wasn’t supposed to be a test about the label “subject guide.” However, less than 30% clicked on the Subject Guides link; the majority used some sort of search.

Here you can see the places people went to on our home page most (highlighted in red), next frequently (in orange) and just once (yellow).

When people used our site search, they had little problem finding the guide (although a typo stymied one person). However, a lot of participants used our Summon search. I think there are a couple of reasons for this:

• Students didn’t know what a subject guide was and so looked for guides the way they look for articles, books, etc.
• Students think the Summon search box is for everything

Of the 6 students who did click on the Subject Guides link:

• 2 used broad subjects (and neither was successful with this strategy)
• 2 used narrow subjects (both were successful)
• 1 used the A-Z list (with success)
• 1 used the autocomplete search (with success)

One person thought that she couldn’t possibly find the Ottawa guide under “Subject Guides” because she thought those were only for courses. I found this very interesting because a number of our subject guides do not map directly to courses.

The poor performance of the broad subjects on the subject guide page is an issue and Web Committee will look at how we might address that. Making our site search more forgiving of typos is also going to move up the to-do list. But I think the biggest takeaway is that we really have to figure out how to get our guides indexed in Summon.

Advertisement

Weekly user tests: Login revisited

Back in Week 3, we did a test to see if people knew to use our Login button to get into their Library account to renew books, see fines, etc. It was not a great success in terms of recognizing that “Login” got you where you needed to go.

The Fabulous Kevin Bowrin and I thought that maybe people would respond better to a button labeled “My Account,” since some of the comments we received during the test used the word “account.” I altered the button, took a screenshot and headed out to the quad with our first staff volunteer (yay Jane!).

We limited our question to one: “Imagine you’ve taken books out of the library and you want to renew them. Where would you go to do that?” We consistently heard “Services” and “Borrowing” (which is under “Services”) and one person said they’d go to the catalogue. No one mentioned “My Account.” Sigh.

But, every section they mentioned would (eventually) get them to where they needed to be. So do we have a problem? Web Committee has decided we don’t. Login does get used, our analytics tell us that. The button was always meant to be a shortcut, and if not everyone goes to the shortcut but manages to find what they need, then maybe “My Account” is a solution in search of a problem. And I hate those. So Login will stay.

For now.

Weekly user tests: Searching

During the Week 5 “explain the library website” test, one participant had a lot to say about Summon. He wasn’t happy about the change, although he did say that “it’s good, but it’s different and I’m an old man” (he was in his mid-20s at most). I wondered if he would prefer a newer style of search results page — the bento box (see this Musings about librarianship blog post for an overview with screen shots). I showed him a site with bento box results and asked him if he found this better than our Summon results. His reaction was immediate: “No, not in the slightest; this is more annoying… Maybe it’s just because it’s different, but I don’t like it.”

I found this interesting because a number of libraries have moved to bento box results (e.g. UCalgary, Simon Fraser, Duke, Stanford, Princeton, NCSU) and I find the concept quite compelling. Was this guy, who admittedly did not like change, an anomaly?

So for Week 6, not only did we look at how students were using our Summon search, but we asked them to do the same search on one or two of the bento box sites to see if bento worked better. Chocolate? Check. Off to the lobby!

Searching Summon

Overall, students had very little difficulty with searching Summon. Although in our “explain the library website” tests, students mentioned Boolean a lot, only one of our four participants used a Boolean search in Summon. Two used phrases of four or more words. All of them saw the filtering options on the left side and appreciated  that  they could narrow their searches in this way.

A few problems we noted:

• No clear ability to limit to eBooks
• Confusing to have Hold/Reserve button appear for Reserves items
• SFX link resolver interface very problematic (moving to 360Link should help)

Bento Box Results

Of the five students who saw the bento box results (4 this week, 1 the previous week), three disliked the whole concept instantly. Their reaction was immediate and visceral: “Oh, oh, this is confusing, I don’t like this.” “Uh, um, I don’t really understand what it’s telling me.” “Ohhhh… Not a big fan.”  I edited the audio of these reactions together and it’s a symphony of disgust.

Those who quite liked the initial results screen were then unimpressed when they clicked “more results” or the Articles heading and were brought to a discovery-style screen. They both said they would have preferred going to the discovery screen directly, one continuing “because it already has all the filters and everything here.” They did not see any value in separating out Articles from Books when Summon allowed them to do that with the facets on the left. And they weren’t interested in any of the other bento options beyond Articles and Books.

I was surprised that none of the students liked the bento box approach. I still think there are use cases where bento would be an improvement, but I don’t think it balances the negative experience for the majority of our users who really do just want articles and/or books.

One thing that makes me really happy about all of this was that we saved a lot of time and resources by testing the implementation of the concept at other libraries instead of starting to develop it ourselves. It’s certainly possible that our five random students were anomalies and that bento would be a big improvement for our users. But five out of five negative reactions is enough to let us decide to put our focus on other projects. We may well revisit bento again, but only after I hear cries of delight drown out the cries of dismay.

Weekly user tests: Explain our website

It seems to me that many user tests (mostly usability tests) have students doing very specific tasks that they may never do in real life. Of course user testing is artificial, and usability testing generally needs to be task-based, but is it useful to evaluate our sites with people who don’t care about the task they’re performing?

I thought one way to elicit information about tasks that meant something to students would be to ask them to explain the site to a peer. This way, maybe we could get a sense of what sections of the site they saw as particularly useful and which were not even on the radar.

Armed with chocolate, we hit the lobby of the library to ask students for about 5 minutes of their time. We asked them to pretend I was a first year student and explain the library website to me — what was useful, what wasn’t, what I could safely ignore. The plan was then to ask what task they do most often on the library website and walk us through that task, and then to ask what task they found most confusing and walk us through that.

I’m going to pause here and say that I pretested our script and it worked really well in the pretest.

We did this test in Weeks 4 & 5, with 10 participants total, and what we found was that the students really only used the website to search. They searched the catalogue, Summon and databases, but that was about it. Some signed into Ares for their course reserves. A few used subject guides, but exclusively as a source for which databases to search. Did they mention any actual content that a library staff member wrote? Just once: booking a group study room.

So, the question about the task they do most often? Well, they had already answered it: they search. Most confusing task? When search doesn’t work.

There were a few interesting things that did come up (*list below). But I had been hoping to find some big pain points, areas of friction that we could attack. I guess it’s good news that students generally expressed satisfaction. But maybe we just asked the wrong questions. “Explain” is probably the wrong word. It probably worked well in the pretest because I pretested with a student employee. Who explains the website as part of his job. Ah, hindsight.

Now that I think about it more, I’m not surprised that search was all that came up. A huge percentage of use of the library website is search. Asking random students what task they perform on the site, is more than likely going to bring up search. So the next test: How well is search working? Could we do better?

 

*A few interesting things that did come up in these two tests:

• Many students told us things they had obviously learned through instruction, but sometimes they got things a little wrong (use the catalogue for journal articles and Summon for books; in a Boolean search you put AND in brackets; you have to use Boolean to combine words in Summon; you have to use the Summon advanced search to limit by date; )
• Some students find eBooks very inconvenient to use (“It’s like a scanned, older-than-pdf kind of thing”). A few prefer print books to eBooks.
• Moving to MyCarletonOne login made login less confusing
• Adding the action button to the group study room page made that page less confusing

UXCamp 2014

This weekend’s UXCamp Ottawa provided much food for thought. (So many great speakers!) My notes are not at all complete. This is just the stuff that resonated most, and what it made me think about:

• Lou Rosenfeld’s talk on breaking down silos resonated a lot. I’d read the A List Apart article that the talk was largely based on, but hearing him speak made things stick a bit more, or maybe I just had the space to think about them more.
  • I think we do okay with balancing qualitative and quantitative research, but I should probably document that and see if there’s any way to formalize the cadence of our research. We have tried to plan testing to follow the cadence of the academic year, but again, documenting it would be A Good Thing.
  • Ban words that impede conversation. Words can make people shut down (In the next presentation, Abby Covert called it “linguistic insecurity”). We should figure out if we (Web Committee) use any words like that in our communications with staff.
  • I should throw the results of our various “problems” and feedback forms (and maybe other stuff?) into Evernote so they’re searchable. I’m sure there are commonalities that we’re not seeing.
  • Great question: “If you were going to build the organization’s decision-making brain from scratch, what would it look like?”
• Abby Covert’s talk covered some similar themes, but I felt like she was shining a light on some of the big issues we have in libraries. She was awesome.
  • As well as talking about banning certain words in meetings (she uses a gym whistle), she spoke about how to bridge gaps between people and groups by having them refer to objects (like mental maps). When we talk about something concrete, we can find it easier to speak each other’s language.
  • Not just banning words in meetings, it could be useful for us to be specific about what words we don’t use on our website. The New York Times has a list of words they don’t use, and it speaks volumes about their culture. Can you imagine if the library could agree on what words we don’t use? I got excited about the idea of bringing students into that conversation too (as if it wasn’t a big enough task already!). Hmmm….
  • Again, related, the structure of our site is rhetoric. What is our rhetoric? Since it’s there, we should make some conscious decisions about it.
  • Diagrams can be really great conversation starters. Or even whole conversations. I should really sketch more. And encourage others to do it too.
• Related to that last point, Konrad Sauer talked about sketching too. He said that the important thing is that a sketch take you back to the idea you were having when you drew the sketch. I like that.
  • Konrad also spoke about the importance of the personal in the tools he uses; he feels a connection to the person who used it before him and the person who made it. Where is the personal in the library’s tools? Is there any way we can help people feel a connection? And what kind of connection would it be?
• Steve Krug is simply great. And part of what makes him great is that he make things sound so simple.
  • If Steve Krug uses a script faithfully, I should maybe start to use a script faithfully. I know that I use words like “opinion” and “feedback” when I’m doing testing. I shouldn’t and I will try to avoid those in future.
  • It made happy that Steve uses Camtasia in his tests. I was feeling like I was being lazy using what we had on hand.
  • “Why waste time having people point out the problems we already know about?” Because they’ll also find the problems you *don’t* yet know about.
  • The shared experience of observing tests is important for people in your organization. I have to think more about how to do that. Maybe playing the recordings for a crowd would suffice?
  • Always buy the good snacks.
  • Beware of low-hanging fruit; focus ruthlessly on the most serious problems first.
  • Try to find the smallest change you can make that might solve the observed problem.
  • Recommended books: “Letting Go of the Words” by Janice Redish (this is good timing for advice about a book on web writing!), and “The Moderator’s Survival Guide” by Donna Tedesco.
• Jonas Woost was a lovely, slideless speaker, talking about his work at CBCMusic.ca.
  • His mention of the tension between responding to the legacy users and building a new user base resonated with me. Sounded similar to the tension between expert and naive users in the library.
  • Another “sounds familiar” moment was the difference between “What do I want the users to do?” and “What do the users want to do?”
  • It’s always useful to be reminded that you’re going to have to stop doing some things.
  • We can’t always innovate; sometimes we have to compromise. It’s okay.
• Lisa Fast reminded me of some things I forgot that I knew (and some things I didn’t know)
  • The principles that we use to make certain design elements stand out (boxes particularly) can also make them invisible – “that stuff in the coloured box over there looks different… so it can’t possibly be related to the thing I’m looking for.” Ack!
  • But we can use those principles again (colour, proximity, shape) to bring the design back together.
  • I should get around to reading “100 Things Every Designer Should Know About People” one of these days.
• Kim Goodwin essentially gave a clinic on how to do a keynote speech. Lots of good stuff.
  • We have to understand our own organizational context – particularly our organizational values – in order to figure out or design a way to do our work within it.
  • For a lot of (most?) people, change means a loss of some kind. When we are recommending some sort of change, we need to think about that and address it. It helps to:
    • Create dissatisfaction with the status quo (user tests can help with this)
    • Be clear about what the change is, and have that communication come from all levels, not top-down.
    • (there was a third point, but I missed writing it down – I’ll try to find it later and update)
  • Having explicit principles and values that the organization shares can help with all of this. At a company Kim works with, they have specific design principles and then play “design principles bingo” to try to find violations of those principles on their website.
    • I wondered if we could come up with web writing principles and play bingo with them!

On a personal note, Kim’s point that it can be very uncomfortable when your organization’s values don’t match your own hit close to home. That was my experience at my previous job, and it makes me feel immensely lucky to be in a place where there seems to be a really good match. I feel even luckier that I believe I can bring the things I learned this weekend to my colleagues and we can work on some of them together.

Weekly user testing: Login

One of the issues that was hanging around from the last redesign of the website was whether users understood what the Login button in the upper right corner did. Analytics showed it got use, but we weren’t sure if it was intuitive to first-time users or if people needed to be taught to use it.

Time for another guerrilla test!

The Login button takes people to a list of services that they can access, such as their library account, RefWorks, and RACER (ILL). We decided to focus on whether people understood they could use Login to access their library account to renew books, see fines, etc.

Again, we printed out the main page of the library site and headed out to the quad. We talked to about a dozen people and only two would use the Login button. Some of the strategies mentioned would tell people how to access their library account, so it wasn’t a wash. But we’re left with questions about that Login button because it’s obvious that it’s not, well, obvious.

Weekly user tests: Library hours

This fall, Web Committee planned to do weekly user testing. We started on the week of September 22 and have only missed one week entirely (last week’s was postponed to this week, so we’re doing two rounds this week).

The plan was to alternate between very short, guerrilla-style tests and longer tests. This way, we could take problems that arose in the longer tests and see if we could iterate a solution using the short tests. So far, we’ve done three short tests followed by three longer tests. So not much iteration. Yet.

The first two weeks’ tests looked at a possible change to the Library hours information on the main page.

We thought our main page was missing a place where we could promote services or resources for longer than would normally happen with a news item. We wanted to create a Spotlight for these kinds of things. When we moved to Drupal 7 we increased the width of the content portion of the page. Our Hours seemed to use a lot of space, particularly on smaller screens, and we thought we could easily condense it to make room for the Spotlight.

Week one

The fabulous Kevin Bowrin mocked up a design for the main-page-with-Spotlight. We printed it off and, clipboard in hand, headed for the quad for some guerrilla testing.

“Can you show us where you would find the library hours for today? Can you show us where you would go to find the hours for tomorrow?” Those were our two questions. And a lot of people couldn’t find today’s hours. Even fewer could find tomorrow’s.

This was not good.

To my eye, it was a classic case of ad blindness with a generous dollop of just too much content. The Spotlight was predominantly image-based, and the hours were immediately above it. Everything was to the right of our main Summon search box. People’s eyes slid over the whole section and didn’t catch on the hours at all.

More troubling was that people who saw the hours weren’t understanding that they could click “Library Hours” to get all of our hours. True, the link wasn’t underlined, but none of the links on the site are underlined. Time for another test!

Week 2

In week 2 we decided to test our current page design. We asked “Can you show us where you would find today’s hours? Where would you go to find out when we’re open this weekend, over Thanksgiving?” Everyone found today’s hours. Only one person found where to go to find the weekend hours.

Again, this was not good.

The problem seems to be that we have a header for Library Hours, which is also a link. After today’s hours we have a smaller link for “service hours.” This is where most people said they would go to find our weekend hours. The service hours link takes you to a page that lists our various service points; you must click on a service point to get to their hours calendar.

I looked at our analytics for that page. Most people came to it from the main page, but almost nobody clicked on a service point when they got there. Most people either went to the main Hours page or back to the main page.

Quick solution: we kept everything the way it was, but changed the “service hours” link so that it went to the main Hours page. The services points are also listed on that page, so anyone who wants service hours can still access them. But the majority of people will get hours calendar they were looking for.

Oh, and the Spotlight? We’ve decided to try to implement it in another way. After more testing of course.

Retrospective post: Revamping our 404 page

When I started at Carleton back in 2013, one of the first things I did was make a more friendly 404 page. The “page not found” message at the time looked like this:

Not very helpful, and it didn’t even have our footer menus included. So, I added what I thought were useful options if you happen to hit a 404 page – search, index, help pages, feedback to tell us we messed up and deleted something important. Plus a fabulous image created by my fabulous colleague Kevin Bowrin. Since I was new to Carleton, I didn’t get the whole raven thing (Carleton sports teams are the Ravens) and dubbed this lovely guy the “Oh no crow”:

Less than two weeks into the job and I knew I was going to love working here.