Our work is never done: evaluation and iteration for a new audio guide
AbstractIn December 2015, the British Museum launched a new audio guide. Based on extensive generative user research (Mannion, Sabiescu, & Robinson, 2015), its design and functionality challenge the traditional notion of an audio guide and provide multiple ways to engage with content, wayfinding, and object information. We embrace the notion of iterative product development and wanted to improve the audio guide product that has now been "on the market" for over six months. Too often, the concept of "iteration" is used in reference to technological development, with user research limited to a/b testing of designs for immediate implementation. We took a broader view and chose to spend significant time and budget conducting evaluative user research to understand what we did (and didn’t) get right in the first version of this new audio guide. In this paper, we explore and share four key points: our long-term goals for undertaking this research; our methodology for conducting it; lessons learned and advice for other institutions; and, our revised strategy for the digital product. Through our most recent user research, we were able to observe significantly different visitor expectations for and use of the audio guides amongst three linguistic groups, giving us the opportunity to consider how we may change the digital experience to accommodate these varied expectations and experiences. Our conclusions will be of interest to any cultural organization building mobile digital products, and especially those serving a multilingual population.
Keywords: audio guide, multimedia guide, museum, mobile device, research, strategy
In December 2015, the British Museum released a new audio guide, replacing the museum’s old multimedia guide, a product that launched in 2009. Based on extensive generative user research (Mannion, Sabiescu, & Robinson, 2015), its design and functionality challenge the traditional notion of an audio guide and provide multiple ways to engage with content, wayfinding, and object information. We embrace the notion of iterative product development, and wanted to improve the audio guide product that has been “on the market” for over six months. Too often, the concept of “iteration” is used in reference to technological development, with user research limited to a/b testing of designs for immediate implementation. We took a broader view and chose to spend significant time and budget conducting evaluative user research to understand what we did (and didn’t) get right in the first version of this new audio guide.
In this paper, we explore and share four key points: our long-term goals for undertaking this research; our methodology for conducting it; lessons learned and advice for other institutions; and, our revised strategy for the digital product. Through our most recent user research, we were able to observe significantly different visitor expectations for, and use of, the audio guides amongst three linguistic groups, and we now have an opportunity to consider how we may change the digital experience to accommodate these varied expectations and experiences. The potential of mobile technologies expands almost weekly, as existing hardware and software are refined and new product brought to market. User research and feedback should steer any adaptation of such new technologies in the museum; we will highlight the key takeaways which will guide our course in the future.
We owe a debt of gratitude to our colleagues whose original research was summarized in the aforementioned MW2015 paper. That effort helped to create an excellent and generally well-received audio guide product which we can only make better through our additional research and development. Mannion et al. framed their synthesis in terms of overall digital strategy at the British Museum—a vision we will return to at the end of this paper.
The audio guide at the British Museum consists of a bespoke application running on HTC Desire 626 smartphones. Initially launched in December 2015, the application had seen several minor iterations before summer 2016. Information can be accessed in ten languages, as well as British Sign Language, and users choose how to explore the collection: by taking a tour (complete with step-by-step directions between objects), browsing object lists by culture area, or wandering the galleries and typing in stop numbers found on object cases. Object “stops” include visual content to accompany the audio track, and many contain additional information such as the object’s findspot, accession history, or a secondary audio track. We also offer a family guide on the same devices; this consists of an interactive game which a group can play. In this paper, we will focus on the main audio guide offering.
The initial design of this app grew out of the generative research and the digital strategy of the museum. Research indicated that visitors had a preconceived idea of an “audio guide” and therefore felt it would restrict their experience to a single, pre-defined path. Visitors wanted different ways to explore the collection and were conscious of the time constraints, perceived or real, of their visit. Our digital strategy emphasizes reach, relationships, and revenue: we want to get this product into the hands of as many visitors as possible, not just to increase the sustainability of our digital offerings, but also because we aim to significantly enhance the experience of visitors who use it and to help them develop a deeper connection with the museum collection. We feel that we are making progress towards these goals, as pickup of this paid product increased from around 2% of all museum visitors before December 2015 to over 5% in August 2016.
In the first seven months after launch, the audio guide product faced several challenges. Some related to the software: for example, analytics weren’t properly integrated at launch so it was difficult to understand user behavior. We knew, from observation and feedback from the retail staff, that parts of the application confused users and were not as obvious as we had conceived them to be. Moreover, some minor features had not been completed for the original launch date. Other issues involved the hardware. Early on, the SD cards did not fit the devices properly, but this fault was very difficult to diagnose. In June 2016, some devices started to “swell”; over the summer, hundreds of devices manifested what we came to understand were swollen batteries. Given the high-profile battery issues in an array of mobile devices in 2016, we have been especially concerned with identifying the root cause of this issue and addressing it; we continue to work towards a resolution with our hardware supplier.
The December 2015 product launch was never meant to be the final version of the British Museum audio guide. Our digital team and strategy embraces iteration, and the challenges just outlined provided plenty of fodder for potential improvements. Knowing that August 2016 would be a busy month with many visitors, we wanted to conduct user research at that time and to use the insights gathered to make usability improvements to the app itself (as well as the general audio guide experience at the museum), while also addressing some of the technical challenges identified in the first seven months of the product’s life.
The core team that collaborated on this research consisted of a product manager, a user researcher, and a junior product manager (who has since moved into user experience design). We consulted other stakeholders to gather input and share plans or progress at multiple points. We defined the questions we hoped to answer, which broadly fell into two categories: product usability and general service design. On that basis, our user researcher defined a program of inquiry which relied on the core methodological tenets which we had agreed upon. Chief among these was our conviction that we ought to try to better understand the potential differences between visitors from different parts of the world, who might have dramatically varied expectations of, and needs for, a mobile museum product.
We chose four languages (out of the ten in which the audio guide is available) which best represented different parts of the world and, therefore, different sets of expectations. English-language speakers, largely from Britain and the USA, consistently make up a significant proportion of overall visitors to the British Museum and also of audio guide users. While the guide is offered in Chinese, Japanese, and Korean, Chinese tourists represent a visitor segment that rivals the Anglophone and has been growing rapidly. Therefore we chose Chinese as the second language for this project. To represent Western Europe, we considered the historical data for German, French, Italian, and Spanish pickup of the guides; Italian and Spanish had significantly more—but roughly equal between them—audio guide pickup numbers, and we ended up choosing Italian due in part to our internal skill set and the availability of native speakers. Finally, we wanted to investigate other European visitors who might not speak a Romance language or German; we chose Russian as the final language. We worked with trusted native speakers whom we trained in the basics of user research to translate all research materials and perform analysis.
To achieve a wide spread of qualitative and quantitative data across these four user segments, we developed surveys and employed short interviews before and after use of the audio guide. Unmoderated surveys (in the four languages) were displayed on temporary kiosks in the queue for guides and near the return area. The moderated surveys, or short interviews, took from five to ten minutes; for these, we approached people as they stood in the queue or just after they returned their devices. We also conducted longer usability interviews, in which two researchers followed the subject(s) for 30-45 minutes as they used the product naturally, asking clarification questions and observing pain points with the technology itself. Due to the extremely low numbers of Russian-speaking visitors, these usability interviews only took place in English, Chinese, and Italian.
Overall, we spent three weeks on-the-ground with audio guide users, as well as several weeks preparing all materials and training our native speakers, plus analysis of the data afterward. Our data set consists of 191 unmoderated surveys, 110 short interviews, and 17 longer usability interviews. We also systematically observed the audio guide retail desk and space around it according to a rubric, and followed audio guide users at a distance with no interference to observe their visits and experience with the product.
Table 1: participant totals for unmoderated (kiosk) surveys
|During use (usability)||5||6||6||17|
Table 2: participant totals for moderated interviews
Analysis & trends
Certain trends appeared across the entire set of users with whom we interacted. We will address these first, and then move into the arguably more interesting observations we were able to make about the different segments loosely defined by language. While we appreciate the framework proposed by our colleagues (Mannion, Sabiescu, & Robinson, 2015), for understanding the factors which lead someone to take an audio guide, we have not framed our findings exclusively in those terms. Certain themes from this framework will nevertheless recur in the following analysis.
We confirmed or expanded upon certain key findings that Mannion, et al. initially identified and upon which the design of this product is predicated. We wish to underscore the importance of confirming previous work in this way: a major pitfall of product development based on user research can be the continuous pursuit of new information and development of new features without seeking to understand whether the product actually addressed user needs as intended. Mannion noted that visitors often come to the museum with specific expectations about the highlight objects they want to see; we observed the same behavior almost universally, and noted that aspects of the audio guide’s design allowed users to pursue such an agenda. Another key earlier finding, which our work reinforced, was that users want to access additional information about the collection (in their own language, if they are not native English speakers). As our proportion of non-Anglophone visitors continues to grow, we need to continue to be conscious of this motivation and find a scalable solution to provide more content in more languages.
Wayfinding continues to challenge nearly all audio guide users. We observed this on a number of levels: firstly, visitors struggled to orient themselves from the audio guide desk in the museum’s Great Court. This large space houses maps (in English only) on large plinths, most of which are not directly visible from the audio guide retail desk. The audio guide app itself contains several wayfinding aids, from Google indoor maps to step-by-step instructions within tours; but users almost universally missed this in-app functionality (and the few who noticed it experienced some issues loading the indoor maps due to weak Wi-Fi connectivity in the Great Court). Users’ successful navigation to the first object they wished to see largely depended on whether they located a printed map, of which the museum offers several, and used it in conjunction with the audio guide. The intention of putting wayfinding within the app was to render such extra aids unnecessary; we realized that this approach simply hadn’t worked. We found that users weren’t necessarily expecting to be able to use the device as a wayfinding tool; hence, they didn’t necessarily find it in the audio guide. We concluded that wayfinding within the app is useful, but not expected. Therefore, we need to understand how it can be improved and more effectively showcase the features which address this need.
Usability of the app itself varied significantly between different types of users. We will return to some cultural differences which seemed to affect user interaction with the audio guide. In general, older users—particularly the 35% respondents over the age of 40—were overwhelmed by the amount of functionality in the app. Younger users, perhaps more used to smartphones, found the app more intuitive; in mixed-age groups, younger users (including children) helped the older members of the group navigate to tours or stops. In fact, older users sometimes expressed a wish for a certain type of functionality in the app which actually did exist. For example, they would often embark on a pre-defined tour. Whilst on the tour, they would see another audio guide stop in a gallery which interested them but was not included on the tour. Upon probing, we would find that when they wanted to access a stop outside the tour, they were too nervous that they would lose their place on the tour to explore the app and find the functionality they desired (i.e., a keypad which allows them to type in any stop, and the option to return to the tour via a persistent tour menu).
The vast majority of users interviewed expressed a desire for more content, although they found the content provided to be of high quality. Some wished for more depth about specific objects; many wanted to know more contextual information about the collection in a given room. Content production presents a challenge, as outlined above, due to the number of languages in which we offer the audio guide. We will likely never be able to offer an audio guide stop for every item on display at the museum unless we radically change our approach to these. In the short term, we will undertake targeted content augmentation and will discuss both the short and long-term strategy below.
Specific usability considerations which will likely be of interest to other institutions include our finding that museum terminology confused audioguide users. In particular, the concept of the “collection” did not resonate—in any language—with them, nor did they think of the items in the museum as “objects.” The audio guide home screen displayed three options (“View Highlights,” “Take a Tour,” and “Explore the Collection”), based on what we initially understood users’ needs and desires to be, but the way these options were articulated led to increased confusion. Users did not understand the difference between “View Highlights” and “Take a Tour”—the former is a list of objects without any wayfinding or other unifying theme, while the latter includes a selection of tours including a “Top Ten” option. Users expected “View Highlights” to provide a tour of museum highlights, effectively the same as the “Top Ten” tour, and grew frustrated that there was no clear logical arrangement of the objects within that section. We concluded that “collection” was not an effective way to express the idea of exploring the whole museum, and that we needed to reconsider how we were highlighting famous objects.
While we saw these trends across all users, we also found certain key differences between Asian and European audiences in particular, and between the Italian and Anglophone groups as well. The previous research focused relatively less on cultural differences and more on general reasons to take an audio guide, but did note the importance of accessing information in one’s own language. One other specific behavior, previously identified and confirmed anew by this research, was Chinese users’ tendency to look first for objects from China in the museum—often before turning to other famous museum highlights. This proclivity introduced a special challenge to our research as the Chinese galleries remain closed for refurbishment, leading to many disappointed Chinese users whose natural behavior was thus altered. European and Anglophone users, by contrast, tended to seek out famous objects—the Rosetta Stone, mummies, or Parthenon sculptures—as the prime motivators of their visits.
Attitudes towards mobile technology varied noticeably between the linguistic groups. We hoped to better understand what impact the smartphone-based app had on the user experience, as it contrasts sharply with the purpose-built, receiver-like audio guide devices used at many other museums and attractions. Chinese users generally found the app to be intuitive and spent the least amount of time struggling with functionality. This level of comfort likely reflects both the overall saturation of the Chinese market with Android-based smartphone devices (Poushter, 2016), and the probability that tourists in London from China have sufficient income to own a smartphone. Additionally, 72% of Chinese tourists expressed interest in using their own devices to access audio guide content and functionality (compared to 66% of users overall); anecdotally, we learned that they often bring external batteries to power personal devices through long days of sightseeing.
In contrast, Italian-speaking users generally missed most of the extra functionality in the audio guide app and expected it to work like a “traditional” audio guide with numeric inputs for stops and little else. We observed many of these users simultaneously using personal smartphones, so a lack of familiarity with the technology does not explain their reluctance to explore the app. We suspect this type of use stems from two related causes. On the one hand, Italians are used to audio guides, both at home and abroad in Europe, and are culturally primed to take one when available—but they are used to the traditional style of guide. Once they figure out how to achieve their objective of accessing information in Italian on the guide we offer, they generally have little interest in exploring other functionality.
As might be expected, English speakers did not use audio guides primarily to learn about the objects in their own language, but either as an auxiliary to museum labels and signage or because they preferred the audio format. Their expectations tended to revolve around content, expecting extra, hidden, or (in one user’s words) “quirky” information about the objects in the guide. Again, though, we observed a lack of interest in exploring device functionality to meet these expressed needs: while “additional content about objects” was consistently requested, these same users almost universally missed the additional audio content available for many of the popular stops on the guide.
On a final note, almost all guide users would pick up the device when they wanted to find or navigate to another object, but let it hang down and ceased to look at the screen once they located the object about which they were listening—despite the multimedia content available on-screen during stop narration. Upon probing this behavior, we determined that most users preferred accessing additional information in audio format because they can visually focus on the actual objects in the gallery. They do not see graphical multimedia content as a significant additional value to their experience.
The wealth of insights into the audio guide user experience inevitably led to all sorts of ideas for improvement. After a blue-sky brainstorming session of ways in which we might address the pain points discovered through research, we faced the difficult task of deciding which to productize, and when. A useful lens for this task let us consider which challenges actually blocked use of the product or the overall experience in the museum (“I couldn’t figure out where to go for the first stop of this tour”) versus those which were more inconvenient but not truly visit blockers (“I’d like to see a map of the whole museum with all the audio guide stops labelled”). We also identified changes which might not be strictly necessary for usability of the product, but which seemed most likely to satisfy a variety of user desires across linguistic-cultural groups.
To this end, we conceived of two general sets of updates which we plan to complete before summer 2017, when visitor numbers swell for weeks on end, and during which period we prefer not to release software or content updates. The first set focuses on technical aspects of the application, while the second addresses content. Autumn 2016 saw a series of sprints aimed at (a) reducing technical debt left over from previous releases; (b) making incremental usability improvements as identified from the user research; and (c) introducing a new feature aimed at protecting our devices by reducing theft and accidental loss. We will primarily discuss the second point here, but wish to underscore that we saw the reduction of technical debt as worth our time and resources, even at the expense of releasing as many usability updates as we might have liked; the long-term success and stability of the audio guide application depends on having the right infrastructure as well as happy users.
One key insight that greatly affected which usability updates we prioritized was the realization that the audio guide app did possess much of the functionality people wanted—they just couldn’t find it, for one reason or another. This realization helped us think of the updates as a design challenge rather than a significant refactoring of the app itself: how might we better highlight existing functionality and content? Two features stood out as especially ripe for redesign: the home screen, from which users choose their initial path into the museum collection, and the stop detail page, where users can access not just an audio track but other types of additional information about the object on view.
We have already touched on some of the challenges of the home screen, which prompted the user to choose from three options which weren’t clearly-enough differentiated. In particular, we needed to find a way to express the idea of browsing objects without taking a tour: “Explore the Collection” wasn’t working well in any language. Our design team created several options, tested these with museum visitors, and concluded that a scrollable carousel of objects better conveyed the idea of browsing galleries. Showing, rather than describing, this functionality to the user, obviated the need to find better copy.
A major theme of all groups interviewed was a desire for more contextual information, and more content, in general. We are addressing this in the second set of updates, mentioned above. In the shorter term, we have undertaken a set of content updates which aim to even out the coverage of stops so that every permanent gallery has at least one stop. Most importantly, all permanent galleries will have an introduction which provides general information about the collection of objects in that room; for example, whether they are unified by a particular location or excavation, culture, bequest, or time period. Curators are collaborating with us to create these introductions and, in almost all cases, to record the English-language version. We believe these additions will meet visitors’ expressed and observed desire to have additional contextual information about collections of objects; to hear directly from curators; and to have some content with which to orient themselves when they first walk into a gallery before examining specific objects. Of course, we will evaluate whether these content additions have the desired effect next summer in our upcoming user research.
Future vision & recommendations
What comes next, after the content updates planned for early 2017? As discussed above, we did not pursue a solution to every user challenge identified during user research. Service design looms large in our product team core principles, so we will not rush to further software development. Rather, additional innovation for the audio guide product will fit into museum-wide developments (such as the implementation of a new Wi-Fi system, which should solve some of the connectivity-related issues on the audio guide whilst also improving the overall visitor experience in many other ways) and specific improvements to the audio guide acquisition process. The latter include a pilot program to offer groups the ability to pre-book audio guides and pick them up in an alternate location. We’ll also continue to seek more information about the demand for, and viability of, a BYOD offering—at least for certain sets of visitors who are likely to have the personal equipment to take advantage of it. Of course, we also plan more user research to evaluate our progress and identify further or new opportunities.
More generally, we seek to fit the audio guide and future mobile development into a coherent digital experience for visitors, both on- and off-site. Our product team includes a robust user research and user experience component, and the responsibility of that team extends far beyond simply testing prototypes of specific digital products. As they help us continue to understand the many possible visitor journeys at the museum and evaluate the touch points we have with them, we can continue to evolve in our understanding of how mobile, specifically, can enhance that experience before, during, and after a visit for different sorts of visitors from the greater London area or from Beijing.
Hand-in-hand with our service-design-based approach, we continue to pursue a better understanding of our different audiences and how to differentiate digital product offerings to best meet their disparate needs. The size of the Chinese segment render it an especially attractive visitor group with which to begin. In collaboration with our colleagues in marketing, we plan to examine expanded use of WeChat to enhance these visitors’ experiences. Other colleagues on the digital product team will collaborate on a micro-site for Chinese visitors to help orient them to the museum (a special challenge as many come in tour groups and enter via our rear entrance). Within the audio guide, we could make small changes which don’t require much new development: for example, when the Chinese language version is selected, the app could offer the China tour at the top of the tours list or on the home screen. Such small tweaks present a great opportunity for A/B testing to see if incremental improvements positively affect these users’ experiences. While localizing products to many different audiences will likely fall outside the scope of what most museums can reasonably fund or produce, recognizing the largest demographic segments—whether these are linguistic-cultural groups or some other segmentation, such as families—and differentiating mobile products thoughtfully for these segments—will reward the effort. We will continue to share our lessons learned as we implement this strategy over the next years.
Our research has left us with key tenets which ground our thinking as we look ahead to the next three to five years and consider the future of mobile in museums. As the technology available in visitors’ pockets and, potentially, for museums to install or rent increases every month, user needs must remain central to all future product development. Augmented reality, for example, offers the potential to amplify the on-site visitor experience; our research revealed that audio guide users want to interact directly with the objects in our galleries and value audio delivery of content because it allows them to look closely at the objects on display rather than a screen. Museums are starting to use this technology already: the Detroit Institute of the Arts recently announced its collaboration with Google’s Project Tango to bring AR into its galleries. How might we use new technologies in a way that fulfills the needs we have consistently identified and doesn’t get in the way of visitors interacting with the artifacts themselves?
Another lasting insight revealed that users want to explore on their own terms, without being restricted to objects which have audio stops. The generative research which shaped the current audio guide product recognized this, but we must support this need even better as we shape our future roadmap. The British Museum collection presents a massive challenge in terms of the sheer quantity of objects on display: currently, content for new stops must be written, recorded, and produced in ten languages and British Sign Language. The majority of today’s audio guide users do not speak English, and we are conscious that the languages offered do not cover the needs of all visitors. We must find ways to use developing technologies to offer ever more content in ever more languages so that all visitors can use our mobile product offerings and access as much information as they wish. Machine translation between English and many European languages is now highly accurate; major tech companies continue to work towards similar accuracy for translation between non-Western languages and English. Perhaps automated translation will provide one avenue in the future where we can scale the availability of our content for our on-site audience. Whatever the solution, we need to find a way to go beyond the incremental gains of producing new stops on a one-by-one basis and find a way to scale the availability of information by a factor of ten or 100.
Korean Air has generously sponsored the British Museum multimedia guide since 2009, and the renewal of their sponsorship in 2014 made it possible to carry out this project. Special thanks to Agnese Rovati and Yiman Lin for conducting user interviews with us in Italian and Mandarin, answering many localization questions, and helping us assess the data. Our gratitude also goes to our colleague, Ayla Erimer, for working with us on the user research and reviewing multiple drafts of this paper.
British Museum. (2016). Audio Guide: Key Themes and Usability Report. Internal document.
Dent, S. (2017). “Google Tango AR takes you inside a mummy’s sarcophagus.” Engadget. Published January 10, 2017. Consulted January 26, 2017. https://www.engadget.com/2017/01/10/google-tango-ar-takes-you-inside-a-mummys-sarcophagus/
Mannion, S., A. Sabiescu & W. Robinson. (2015). “An audio state of mind: Understanding behaviour around audio guides and visitor media.” MW2015: Museums and the Web 2015. Published February 1, 2015. Consulted January 11, 2017. Available http://mw2015.museumsandtheweb.com/paper/an-audio-state-of-mind-understanding-behviour-around-audio-guides-and-visitor-media/
Poushter, D. (2016). “Smartphone Ownership and Internet Usage Continues to Climb in Emerging Economies.” Pew Research Center. Published February 22, 2016. Consulted January 22, 2017. Available http://www.pewglobal.org/2016/02/22/smartphone-ownership-and-internet-usage-continues-to-climb-in-emerging-economies/
Hudelson, Natalia, Casey Scott-Songin and Han Li. "Our work is never done: evaluation and iteration for a new audio guide." MW17: MW 2017. Published January 28, 2017. Consulted .