User Research: Beautiful Websites?

My University Librarian has asked for a refresh of the library website. He is primarily concerned with the visual design; although he thinks the site meets the practical needs of our users, he would like it to be “beautiful” as well. Eep! I’m not a visual designer. I was a little unsure how to even begin.

I decided to attack this the way we attack other problems: user research! Web Committee created a set of Guiding Principles a few years back (based on Suzanne Chapman’s document). Number one in that list is “Start with user needs & build in assessment” so even though I was having difficulty wrapping my head around a beautiful website as a user need, it made sense to move forward as if it were.

Background

How does one assess a beautiful website? I looked at a whole bunch of library websites to see which stood out as particularly beautiful and then discern what it was that made them so. Let me tell you, “beautiful” is not a word that immediately leaps to mind when I look at library websites. But then I came across one site that made me give a little exclamation of disgust (no, I won’t tell you which one). It was busy, the colours clashed garishly, and it made me want to click away instantly—ugh! Well. We might not be able to design a site that people find beautiful but surely we can design something that doesn’t make people feel disgusted.

I had an idea then to show users a few different websites and ask them how they felt about the sites. Beauty can mean different things to different people, but it does conjure a positive feeling. Coming up with feeling words can be difficult for people, so I thought it might be easier for me to come up with a list they could choose from (overwhelming, calm, inspiring, boring, etc.). Then I decided that it might be better to have users place the sites on a continuum rather than pick a single word for their feeling: is the page more calming or more stressful? Is it more clear or more confusing? I came up with 11 feelings described on a continuum, plus an overall 🙂 to 🙁.

I wasn’t completely confident about this and assumed others had done work in this area, so I did some reading on emotions, aesthetics, and web design. (Emotion and website design from The Encyclopedia of Human-Computer Interaction, 2nd ed.; Aesthetics and preferences of web pages from Behaviour & Information Technology (2000); Assessing dimensions of perceived visual aesthetics of web sites from International Journal of Human-Computer Studies (2004); and Measuring aesthetic emotions: A review of the literature and a new assessment tool from PLOS ONE (2017).) Turns out my method was in line with the research in this area. And although the wording sometimes differed, the 11 feelings I had come up with were all represented. Onward!

There had been some talk of the library website perhaps needing to mirror other Carleton University websites a little more closely. However, there is not uniformity of design across Carleton sites, so I wanted to show users a mix of those sites to get a sense of which designs were most pleasing. I also wanted to show a few different library sites to get a sense of which of those designs were most appealing to our users. I worked with Web Committee to come up with a list of 7 library sites and 5 Carleton sites.

There was no way I was going to ask someone to give us feedback on 12 different websites; I decided a selection of 3 was plenty for one person to work through. Since I was looking mostly for visceral reactions, I didn’t think we needed a lot of people to see each site. If each site was viewed 5 times (with our own library site as a baseline so we could measure improvement of the new design), we needed 30 participants. That was three times what we often see for a single round of UX research, but still doable.

Method

I planned a 10-minute process—longer than our usual processes where we test one or two things—and wanted to compensate students for this much of their time. That fell apart at the last minute and all I had was a box of Halloween mini-chocolates so revamped the process to remove a few pre- and post- questions and cut the number of continuums from 12 to 9 (8 feelings plus the overall positive/negative). That cut the time down to about 5 minutes for most people, and I was comfortable with a 5-minutes-for-chocolate deal. So in the end, these are the continuums we asked people to use to label the sites:

Welcoming ↔ Off-putting
Disorganized ↔ Organized
Clear ↔ Confusing
Up-to-date ↔ Old-fashioned
Calming ↔ Stressful
Useful ↔ Useless
Inspiring ↔ Discouraging
Ugly ↔ Beautiful
🙂 ↔ 🙁

We set up in the lobby of the library and saw 31 people over four time slots (each was 60-90 minutes long). There were 31 participants instead of 30 because the last person came with a friend who also wanted to participate. Happily, the only person to have difficulty understanding what to do was one of these very last people we saw. He had such trouble that if he’d been the first person we’d seen, I likely would have reconsidered the whole exercise. But thankfully everyone else was quick to understand what we wanted.

Most people saw one Carleton site, one library site, and then our own Carleton library site. Because we had more library sites than Carleton sites, a few people saw two library sites then the Carleton library site. I had planned out in advance which participant would see which sites, making sure that each site would be seen the same number of times and not always in the same order. Participants looked at one site at a time on a tablet with a landscape orientation, so the sites looked similar to how they would look on a laptop. They filled out the continuum sheet for one site before looking at the next. They could refer back to the site as they completed the sheet. I had a note-taker on hand to keep track of the sites visited and to record any comments participants made about the sites (most people didn’t say much at all).

Partway through, I discovered a problem with the “Up-to-date / Old-fashioned” continuum. I was trying to get at whether the design felt old and stale or contemporary and up-to-date. But many people assumed we were referring to the information on the site being up-to-date. I thought that using “old-fashioned” rather than “outdated” would mitigate this, but no. So this was not a useful data point.

Usually with these kinds of processes, I have a sense of what we’re learning as we go. But with this one, I had very little idea until I started the analysis. So what did we find?

Results

I had purposely not used a Likert-type scale with numbers or labels on any of the mid-points. This was not quantitative research and I didn’t want users to try to put a number on their feelings. So, when it came time for analysis, I didn’t want to turn the continuum ratings into numbers either. I colour-coded the responses, with dark green corresponding to one end of the continuum, red to the other and yellow for the middle. I used light green and orange for less strong feelings that were still clearly on one side or the other.

In determining what colour to code a mark, I looked at how the person had responded to all three sites. If all their marks were near the extremes, I used light green/orange for any mark tending toward the middle. If all their marks were clustered around the middle, I looked for their outer ranges and coded those as dark green/red (see examples in the image below). In this way, the coding reflected the relative feelings of each person rather than sticking to strict borders. Two marks in the same place on the continuum could be coded differently, depending on how that user had responded overall.

Examples of participants' filled-in continuums
The circled mark on the left was coded light green even though it’s quite close to the end. The circled mark on the right was coded red even though it’s not very close to the end.

Example of data colour-coded in ExcelAfter coding, I looked at the results for the 🙂 ↔ 🙁 continuum to get a sense of the general feeling about each site. I gave them all an overall assessment (bad, ugh, meh, or ok). No site got better than ok because none was rated in the green by everyone who saw it. Then I looked at how often each was coded green, yellow, and red across all the continuums. Unsurprisingly, those results corresponded to my bad/ugh/meh/ok rating; participants’ 🙂 / 🙁 ratings had been reflective of their overall feelings. Our site ended up on the high end of “meh.” However, several participants made sure to say their ratings of our site were likely high because of familiarity, so we are really likely firmly in “meh” territory.

Now that I’d looked at the overall, I wanted to look at each of the continuums. What was our current site doing really well with? I was happy to see that our current site felt Useful and Organized to participants. “Organized” is good because it means that I feel confident about keeping the structure of the site while we change the visual design. What did we need to improve? Participants felt the site was Discouraging and Ugly. “Discouraging” is something I definitely feel motivated to fix! And “Ugly?” Well, it helps me feel better about this project to make the site beautiful. More beautiful at least.

After this, I looked at which sites did well on the aspects we needed to improve. For both the Carleton sites and the library sites, the ones felt to be most Inspiring and Beautiful were the same ones that were rated highly overall. These same sites were most felt to be Welcoming, Clear, and Calming. So these are the aspects that we’ll concentrate on most as we move through our design refresh.

Next Steps

Now, Web Committee will take a closer look at the two library sites and two Carleton sites that had the best feeling and see what specific aspects of those sites we’d like to borrow from. There’s no big time squeeze, as we’re aiming for a spring launch. Lots of time for many design-and-test iterations. I’ll report back as we move forward.

Access 2018: A UX Perspective

I started my Access 2018 conference experience with a meetup of library people interested in UX. There were only five of us, but we had good conversations about Research Ethics Boards and UX research, about being a UX team of one, and about some of the projects we’ll be working on in the coming year. We also chatted about how we would like to communicate more regularly but how difficult it can be to sustain virtual communities. (Canada is BIG. Heck, even Ontario is big.) It was nice to start off the conference with UX friends – old and new – and my focus stayed on the UX side of things throughout the conference so that’s what I want to write about here.

On Day 1, the first post-keynote presentation was all about UX. Eka Grguric talked about her experience one year in as UX Librarian at McGill. She gets brought into projects in her library as a UX consultant, and also supports others doing UX and user research in the library. She also offers training on UX research methods for interested library staff. Her work is a combination of operational and project-based. She gave a bit of detail about two projects and her monthly operational tests to give us a flavour of the range of methods and processes she uses.

Next up was Ken Fujiuchi and Joseph Riggie from Buffalo State College, who talked about Extended Reality, a combination of virtual reality, augmented reality, and mixed reality technologies. They covered a few different topics (slides here), but what stood out for me was their mention of how user experiences will change as new interfaces become possible and there are new ways for people to interact with materials. They specifically mentioned oral histories moving from audio-only files to users being able to interact with a holographic image of a person who can tell stories but also answer questions. What’s good UX for oral history holograms?

A few presentations also focused on what I see as UX for library staff. Juan Denzer spoke about a project being developed by a pair of students he’s supervising that aims to make it easier to manage EXProxy configuration files (which can easily run to thousands of lines). Having tried to troubleshoot stanzas in EXProxy myself, I can definitely see how this could improve the UX for staff. However, as one of my table mates said, adding an application to manage a text file also adds overhead for whoever has to maintain and update that application. Trade-offs!

Ruby Warren from University of Manitoba was fantastic in her description of a project that didn’t quite get off the ground in the six months she’d set aside to complete it. Ruby had seen that distance students weren’t learning how to use the library in the same way in-person students were (e.g. no in-class visits from a librarian). She wanted to find a way to teach some basic IL to these students and thought that an interactive fiction game would be a good thing to try. She had some great lessons learned (including “Don’t Do Everything in the Wrong Order” and “Plan for Apathy”). One of my favourite things about Ruby’s presentation was that she was upfront about her failures, including – as a UX person – not planning for user testing during development. It’s gutsy to get up in front of your peers and say that you forgot a basic tenet of your discipline because you were too excited about a project. So human but so hard. Yay Ruby! Another key takeaway was not underestimating appeal when planning this kind of project. As someone who has a bard time seeing the appeal of library games, I appreciated hearing this. (I believe it’s possible, but I think it’s extremely difficult.) Ruby’s slides are here.

Back to UX for staff (and users too, to some extent), Calvin Mah from Simon Fraser University spoke about his experience archiving their ILS when his library moved from Millennium to Alma. Some kinds of information were not migrated at all, but even the records that were migrated were not trusted by cataloguers; they wanted to be able to go back to the old records and compare. With these two situations – missing information plus untrusted information – it was decided to build an archival copy of the old system. I find this interesting. On the one hand, I can absolutely understand wanting to help staff feel comfortable with the new system by letting them know they still have the old information if they need it; the transition can be more gradual. But Calvin noted that even though the information is getting stale, staff are still relying on it. So perhaps it’s more of a security blanket, and that’s not good. Also, there was a good library nerd laugh when he said that some staff wanted the archival copy to behave like the old system: “Respect the 2nd indicator non-filing characters skip!”

Something I see as having both staff and user UX implications is having contract work in library systems (probably everywhere, but in systems for sure). Bobbi Fox from Harvard has been on many sides of this situation (as a contractor, as a person hiring the contractor, as a team member, as a person cleaning up after a contractor) and detailed many things to consider before, during, and after contract work in library IT. Too often, contract work results in projects that are difficult to maintain after the contractor has gone, if they are even completed at all. I really like that she specifically mentioned thinking about who is providing user support for the thing(s) the contractor is building, as separate from who is going to own/maintain the project going forward. And in talking about documentation, specifying what documentation those user support people need in order to be able to support the users. This will almost always be different documentation that what is required for maintenance. Good docs are vital for maintenance but if people can’t use the thing, there’s not much point in maintaining it!

Nearing the end of the first day was a panel: “When the Digital Divides Us: Reconciling Emerging and Emerged Technologies in Libraries” that looked at disconnects that can happen on both the staff side and the user side when libraries favour emerging (“shiny”) technology. I thought there were some great points made. Monica Rettig at Brock University talked about issues when access services staff are expected to help troubleshoot technology problems; for staff used to a transactional approach to service, with a heavy reliance on policy and procedures, there is a big cultural shift in moving to a troubleshooting approach. Rebecca Laroque from North Bay Public Library wondered about providing 3D printers while she still has users asking for classes on how to use email. Monica noted the importance of core services to users even though they’re aren’t shiny or new; she asked who will be the champion for bathrooms or printers in the library? Krista Godfrey from Memorial University asked whether library technology should be evaluate and assessed in the same way that library collections are? Lots of questions here, but definitely an agreement that a focus on core infrastructure and services may not be exciting but it’s absolutely vital.

Day 2 was a bit lighter on the UX side. Tim Ribaric gave a great presentation on RA21 and the possible implications of it replacing IP authentication for access to electronic resources in libraries. Tim is skeptical about RA21 and believes it is not good news for libraries (one of his theorems about RA21: “We are effed”). His take was very compelling, and from a UX perspective, he is not convinced there is a clear way forward for walk-in users of academic libraries (i.e. users not affiliated with the university or college) to access our subscription-based electronic resources if we move from IP authentication to RA21. I know some academic libraries explicitly exclude walk-in users, but others are mandated to provide access to the general public so we are used to providing guest access and our users are used to having it. Tim has posted his slides if you’re interested in more on this.

Another interesting UX moment was in Autumn Mayes’ lightning talk about working in Digital Scholarship and Digital Humanities. Part of her job had been working in The  Humanities Interdisciplinary Collaboration (THINC) Lab at the University of Guelph. THINC Lab is a members-only space aimed at grad students, postdocs, faculty, etc. who are doing interdisciplinary and digital humanities research. However, they also host events and programs that are open to the larger university population. So Autumn found herself having to tell non-members that they weren’t allowed to use the space, but at the same time was trying to promote events and programs to both members and non-members. She very succinctly described this as “Get out! But come back!” It’s interesting to think about spaces that are alternately exclusionary and open; what is the impact on users when you make a mostly exclusionary space occasionally welcoming? What about when a mostly welcoming space is occasionally exclusionary?

Bill Jones and Ben Rawlins from SUNY Geneseo spoke about their tool OASIS (Openly Available Sources Integrated Search), aimed at improving the discovery of Open Educational Resources (OER) for faculty at their campus and beyond. The tool allows searching and browsing of a curated collection of OER (currently over 160,000 records). It seems like a nice way to increase visibility and improve the UX of finding OER such as open textbooks.

Again in library staff UX, May Yan and MJ Suhonos from Ryerson University talked about how library-specific technologies can be difficult to use and adapt, so they decided to use WordPress as a web platform for a records management project in their library. One thing I found interesting was that the Ryerson library had a Strategic Systems Requirements Review that explicitly says that unless library-specific technology has a big value-add, the preference should be to go outside library technology for solutions. From a UX point of view, this could mean that staff spend less time fighting with clunky library software, both using it and maintaining it.

The last conference presentation of Day 2 reported on the results of UX testing of Open Badges in an institutional repository. Christie Hurrell from the University of Calgary reported that her institution uses quite a number of Open Badges. For this project, the team wondered whether having an Open Badge that demonstrated compliance with an Open Access policy would encourage faculty to deposit their work in the institutional repository. They did a survey, which didn’t show a lot of love for Open Badges in general. Then they did some user testing of their IR (DSpace), to find out whether faculty would add an Open Badge to their work if the option was there. Unfortunately, the option to add an Open Badge was completely lost in the overall process to deposit a work in the IR, which faculty found extremely time-consuming. Since faculty were frustrated with the process in general, it is very unlikely that an Open Badge would provide an incentive to use the IR again.

The conference ended with the Dave Binkley Memorial Lecture, given this year by Monique Woroniak. Monique spoke about “Doing the Work: Settle Libraries and Responsibilities in a Time of Occupation” where the Work is what non-Indigenous people and organizations need to do before trying to work with Indigenous people and organizations. She gave some clear guidelines on, essentially, how to act with empathy and these guidelines can apply to many communities. However, I definitely don’t want to “all lives matter” this. Monique was clearly speaking about Indigenous people, and specifically about her experiences with Indigenous people in Winnipeg. When she spoke of the importance of assessing our capacity before undertaking new work, she included the capacity to build respectful relationships with Indigenous people. Although it can definitely be argued that a capacity to build respectful relationships is useful for UX work, her caution to never over-promise and under-deliver when working with Indigenous people is situated in the Canadian context of settlers over-promising and under-delivering time and time and time again. Sure, we’ll respect this treaty. Sure, we’ll take care of your children. Of course we’re ready for reconciliation. Over-promising and under-delivering is never a great move, but in this context it is particularly toxic. A few other things that stood out for me in Monique’s talk:

  • Listen to the breadth of opinions in the community. Take the time.
  • This is head work and heart work, and, especially, long-haul work.
  • Look to shift the centre of power for not just the big decisions, but the small as well.

If this interest you, Monique’s talk is available to view in its entirety, as are all of the presentations at the conference (they will be split into individual videos for each talk eventually). Monique finished with a lovely quotation from Katherena Vermette‘s poem “new year’s eve 2013” from her 2018 book river woman:

truth is a seed
planted deep

if you want to get it
you have to dig