Use and Impact of Digital in Cultural Heritage: Insights from the Scottish Network of Digital Cultural Resources Evaluation

Maria Economou, University of Glasgow (Hunterian Museum & HATII), UK


The use of digital in the cultural heritage sector continues to expand, whether it takes the form of an online catalogue, mobile application, or social media exchange, increasingly affecting our cultural encounters and shaping our perception of cultural organizations. Cultural and higher education institutions around the world are investing vast resources on digitization and making their collections available online but still know very little about who uses these and how they interact with the data. In order to address this gap, the Scottish Network on Digital Cultural Resources Evaluation (ScotDigiCH) was initiated in January 2015, bringing together academics from different disciplines and professionals from Scotland’s key cultural organizations and interweaving international perspectives. The aim of the network was to investigate how cultural digital resources are used by diverse user groups, how to record their impact, and how to maximize their potential. It integrated different methodologies and perspectives (from digital humanities, computing science, museology, social sciences) and used as a case study the Kelvin Hall Project in Glasgow. The Network has organized a series of four workshops, a knowledge exchange event and an international symposium on evaluating digital cultural resources (EDCR2016). All these activities have certainly given all partners a lot of food-for-thought and are informing all partners’ digital strategies and how evaluation can be integrated to support all core areas of operation form collections documentation to exhibition interpretation and visitor engagement. The paper discusses the lessons learned from the ScotDigiCH research network, particularly the EDCR2016 Symposium, and the wider implications for other researchers and organizations working in digital cultural heritage.

Keywords: evaluation, impact, digital collections, digital heritage


As the use of digital technologies is expanding in the cultural heritage world, so is our need to understand how these tools affect our users and the impact they have. Whether it is in the form of online collections catalogues, social media exchanges, mobile apps or other interactives in displays, digital is reshaping how we engage with collections and exhibition themes, as well as our perception of the cultural organizations themselves. Yet, despite this increased activity of the last few decades, we still know relatively little about how digital is affecting these cultural experiences and what both the short and long-term impact of these digital encounters is.

The wider challenges about the creation, curation and use of digital collections in the cultural sector raise several questions:
• Who uses the digital collections and interpretative materials that museums, libraries and archives spend so many resources to produce?
• How do they use them, where and why?
• How are these resources re-used, re-interpreted, re-integrated in the users’ own projects and digital networks?
• How can cultural organisations record and assess more effectively current digital use so they can plan their future digital strategies?
• How can cultural organisations work more closely with different audiences to create digital collections and resources that are relevant to them and support them as individuals and members of diverse communities?
• How can we record and assess impact and value of digital cultural resources?
• What are the implications of these issues for policy and future strategies?

In order to explore these questions in greater depth, an interdisciplinary research network was set up in Scotland aiming to bring together existing work in the evaluation of digital in the cultural sector and highlight areas for future research and development internationally. The Scottish Network on Digital Cultural Resources Evaluation (ScotDigiCH, was set up in 2015 and run until the end of 2016 with funding from the Royal Society in Edinburgh.

Background to the Scottish Network on Digital Cultural Resources Evaluation (ScotDigiCH)

ScotDigiCH was coordinated by the Humanities Advanced Technology and Information Institute (HATII) at the School of Humanities at the University of Glasgow, UK. The network brought together researchers from different disciplines and professionals from Scotland’s key cultural organisations in order to investigate how cultural digital resources are used by diverse users, how to record their impact on learning, research, and engagement, and how to maximize their potential. It integrated a range of methodologies and perspectives, from digital humanities, computing science, museology, and social sciences.

ScotDigiCH used a case study to focus the examination of the research questions and ground the research themes in the real working practices of the cultural sector with a focused project. This was the Kelvin Hall (KH) Redevelopment in Glasgow (Figure 1), an ambitious collaborative cultural project (Economou, 2014) which has brought into partnership three leading cultural institutions in Scotland: The Hunterian of the University of Glasgow; Glasgow Life with both the Museums, as well as the Sports section of the City of Glasgow; and the Scottish Screen Archive of the National Library of Scotland with the aim of co-locating their collections in one of Glasgow’s landmark historic buildings (opened to the public in autumn 2016), but also the digital records and resources related to these. The project brought together civic, university and national heritage collections for the first time in Scotland and offers new opportunities for object-based learning (in both the physical and digital context), research and engagement with user communities.

ScotDigiCH examined the challenges and questions raised by the Kelvin Hall project and the use of digital technologies to bring together diverse and extensive collections, support in-depth research but also make them more accessible and easy to use for a wide range of users.

More information about the methodology of ScotDigiCH and its activities are described in Economou (2016a; 2016b).

Figure 1: Artist’s impression of the redeveloped Kelvin Hall building, Glasgow, UK

Research themes and activities

Over a period of two years in 2015-16 the ScotDigiCH network organized a) four workshops, b) a knowledge exchange event with cultural heritage professionals in order to diffuse the lessons learned from the network and receive feedback, c) an international symposium (organized in Glasgow at the end of the project in December 2016), and d) a public lecture and an open evening at the newly re-opened Kelvin Hall exploring the digital collections to coincide with the Symposium in December 2016.

There was live tweeting during all network events, while workshops and Symposium presentations and discussions were video-recorded, livestreamed and put up on the ScotDigiCH YouTube channel.

The workshops organized addressed the Network’s key research areas:

Workshop 1: Models for providing digital access: This examined the different models and approaches in providing access to digital resources in Scotland over the past few years and evaluated their successes and failures. It looked at the particular organizational setting, type of collection and targeted user group of each approach and how these affected digital resource design (Figure 2). Representatives from Scottish cultural organisations and initiatives which were instrumental in the digitisation of collections, such as SCRAN (the Scottish Cultural Resources Access Network) (Royan, 1999), the University Museums in Scotland (UMIS) group (UMIS 20114; Macdonald et. al, 2013) and the team which developed Collections Navigator for Glasgow Museums, were invited to reflect on the lessons learned from these projects and how they can help us plan future ones effectively.

Figure 2: Notes from group activity during Workshop 1

Workshop 2: Recording user needs and motivations. Crowdsourcing, co-curation, co-creation in the cultural sector: This workshop (Figure 3) looked at the relationship between cultural heritage institutions and users as central in the design process. The main questions it examined were: How can cultural organizations engage with and actively involve a diverse user base, one that ranges from young children to experienced researchers, in a creative way? What methods are already out there for generating user-informed content and how have institutions then incorporated these new ideas, insights and perspectives into what they do? Often-used phrases, what does crowd-sourcing (Ridge, 2014), co-creation, co-curation and citizen science (Civic Epistemologies, 2015), mean in practice? And perhaps more importantly, what do these engagement strategies produce?

Figure 3: Group activity during Workshop 2, Glasgow, The Lighthouse

 Workshop 3: Evaluating use and impact of digital cultural resources: This workshop focused mainly on evaluation methodologies:

It was a forum for discussing and comparing methodologies that have been used to examine the use of digital resources and their effect on learning, research and engagement with collections. What quantitative and qualitative methods support this type of research, and what tools can be used by cultural workers and scholars? This workshop considered recent developments such as online usage data analysis and eye-tracking technologies, as well as established ethnographic and anthropological approaches. Crucially, it asked participants to reflect on how we can move beyond simply recording how users interact with digital resources to understanding why.

Workshop 4: Moving from impact to value: Recent socio-economic developments have placed increased pressure on higher education and cultural organisations to demonstrate the impact of their digital resources and initiatives. The associated emphasis on quantitative indicators raises concerns about the difficulties of capturing the real value of cultural resources, and the complex and multifaceted ways in which they affect diverse users. Should we move away from recording impact and focus our efforts more on capturing the value of cultural collections? What are the implications for sustainability and strategy in this process? And how can this be implemented in practice, incorporating rich qualitative data but maintaining robustness and ensuring comparability

International symposium on evaluating digital cultural resources (EDCR2016): This was held at the Kelvin Hall (Figure 4) lecture theatre towards the end of the project to bring together the main questions and issues raised from its previous activities and was aimed at both researchers and digital heritage professionals.

Figure 4: Foyer of renovated Kelvin Hall, Glasgow, where EDCR2016 was held

Profile of network participants

Evaluating all these workshops and events, it was obvious that the ScotDigiCH network succeeded in attracting a combination of researchers and cultural heritage, covering different career levels from postgraduate students and researchers to directors and managers of cultural organizations. There was a good mix of ages, specializations, fields and experience in different aspects of digital creating and curation and cultural heritage. A large part of participants were based in Scotland or North England, but the geographical mix had quite an international profile with several European and some international countries represented.

Although the workshops were initially planned for about 30 participants to allow for small group work and discussion, in the end all workshops were oversubscribed and were extended to 40-60 participants, while the conference was fully booked at 150 places, indicating the interest in these issues and the gap in existing research. The lessons learned from the workshops are described at Economou (2016b).

Insights gained from the workshops

Some lessons learned from the diverse experiences of speakers and participants were highlights at most of all of the workshop, while others were more focused on the particular workshop topics. One of the broader ones mentioned at several of the workshops was the importance of collaboration between institutions on both the creation as well as the evaluation of digital. Another one was about the need to really listen to users and bring them onboard early on and throughout the process of designing digital tools and resources.

When referring to crowdsourcing in particular, Mia Ridge (Digital Curator, British Library) gave a memorable quote at Workshop 2 touching at the heart of the user engagement and volunteer management questions for cultural institutions: “Think of it like inviting people to your home…. so don’t lock yourself in the kitchen”. Andrew Nicoll from Historic Environment Scotland, an organization which has set up several crowdsourcing projects (Hamilton et al. 2014; Clari, 2012) gave the advice “Better to start small and evaluate first before spreading to a huge crowd”.

An issue raised at Workshop2 was that of terminology and the differences between crowdsourcing and citizen science, but also co-curation and co-creation as these terms are sometimes confusingly used interchangeably.

There was recognition of the need for new set of skills for managing effectively projects in this area and to combat the wrong impression that they do not cost anything as they are mainly harnessing the power of the crowd, underestimating the resources that need to be committed from the beginning to make them successful. Crowdsourcing projects have a strong impact on organizational culture and mentality, making the staff rethink their priorities and the aims of the organization.

When discussing methodologies for evaluation, it was clear that there needs to be a balance between quantitative and qualitative ones, as there is no single golden evaluation method. It is also important to integrate evaluation in a life-cycle from the beginning of a project and not bring in as an afterthought. At the same time, the workshops raised the need to move from projects to integrate evaluation work in the everyday practice of cultural organizations and to enlist institutional support at all levels.

International Symposium Evaluating Digital Cultural Resouces (EDCR2016)

The Symposium on Evaluating Digital Cultural Resources (EDCR2016) was the major event to mark the end of the ScotDigiCH network and took take place on 12-13 December 2016 at the recently reopened Kelvin Hall (Figure 5). It was aimed at both researchers and cultural heritage professionals, and provided an opportunity to bring together the main issues, questions and findings raised over the course of the network and its previous activities.

Figure 5: EDCR2016 Symposium poster

 The four key research themes of the ScotDigiCH network were reflected in EDCR2016:

  • Models of access to digital collections
  • Crowdsourcing, co-creation, co-curation in digital cultural heritage
  • Evaluating impact and use of digital cultural resources (methodologies, approaches and issues)
  • Moving from impact to value when assessing digital resources

Submissions on the following topics were sought:

  • Curation of digital collections
  • Working with communities in digital cultural heritage
  • Participatory models of work
  • Methods of evaluating digital resources
  • User studies
  • Metrics, webmetrics, infometrics and usage statistics
  • Crowdsourcing and citizen science in cultural heritage
  • Assessing impact and value
  • Social media usage research

The papers accepted after a review process with a large multidisciplinary international Programme Committee were in the following key areas:

  • Digital collections, users and communities
  • Evaluating the use of heritage visualizations and VR
  • Evaluating digital in museum exhibitions
  • Digital CH – broader perspectives: Funder’s, institutional, organizational
  • Archival collections & audiences
  • Designing and evaluating digital narratives in museums
  • Creation, use and evaluation of digital cultural data

EDCR2016 included also poster pecha kucha style presentations (Figure 6) and a workshop on Using the Europeana Impact Framework to conceive Performance Indicators organized by Harry Verwayen, Deputy Director (Management) of Europeana (, Marco De Niet (Director of the DEN Foundation, the Dutch knowledge institute for digital heritage, and Lorna Hughes (Professor of Digital Humanities at HATII, University of Glasgow).

Figure 6: Thanking the presenters of the EDCR2016 Symposium pecha kucha session

Europeana uses the balance-value-impact model proposed by Simon Tanner (2012) from Kings College London and the workshop addressed in practice the challenges of untangling what meaning these three terms take within different cultural organizations.

At the end of the first day of EDCR2016, Dr Mark O’Neill, Director of Policy & Research at Glasgow Life, and well known internationally in the cultural sector) gave an inspiring and thought-provoking public talk to delegates and local community groups and members of the public on ‘What are Museums for?’, examining the idea of museums, collections from the conception of museums until today and advocating a culture of openness and access if museums are to achieve their potential in society.

The lessons from not only EDCR2016, but the whole work of the ScotDigiCH network are informing the evaluation framework that we are developing at The University of Glasgow (HATII & The Hunterian) as for the EU H2020 EMOTIVE project on emotionally-resonating digital storytelling in cultural heritage settings.

Wider issues for the cultural heritage community arising from EDCR16 and ScotDigiCH

EDCR16 brought to the fore the need for long-term evaluation, and to broaden our perspective from only examining the impact of digital applications on visitors’ or users’ experience in the short-term to trying to capture what the impact will be in a few or even several years from now.

As Marco De Niet (DEN, Digital Heritage Netherlands Foundation) reminded us from the data collected from the ENUMERATE project, only 10% of European cultural heritage collections have been digitized to date. It is evident that a lot of work remains to be done in this direction, both for digitizing cultural heritage collections, but also for better use of the material already digitized, as well as for making the evaluation process more effective.

Both the Symposium and the ScotDigiCH network as a whole highlighted that there is an important gap in understanding better who uses digital collections and tools in the cultural sector, how these are being used and what the impact and value of these interactions are. More research and work is needed to investigate these questions. ScotDigiCH showed that there is no single golden methodology or model to be followed in this direction and that collaboration among cultural heritage organisations and researchers is crucial.

Despite the breadth and depth of digital engagement of cultural organisations and the immense variety of approaches and scale, some of the fundamental questions which still need to be answered remain the same. Sharing honestly and openly failures and frustrations, as well as successes is important to help this growing community of practice move forward. Working collaboratively not only with other organisations and partners but also with end users is also one of the major shifts of the last few years. This requires a shift in our thinking and working practices but also training and policy. There are also very important policy implications related to questions of impact and value of digital resources. In order to address these appropriately it is important to open the dialogue, the way ScotDigiCH started, weaving Scottish and international perspectives, about the best way to record these but also the hidden agendas that are pushing cultural organisations to prove them.



Thanks are due to all the network partners from the University of Glasgow (both at HATII, School of Humanities and at The Hunterian), Glasgow Museums at Glasgow Life, the Moving Image Archive of the National Library of Scotland, and the Department of Computing and Information Sciences of the University of Strathclyde, as well as the ScotDigiCH Research Assistant Dr Rosie Spooner, and to all workshop and EDCR16 Symposium presenters, participants, communication fellows, and volunteers who helped make them a success. The Scottish Network for Digital Cultural Resources Evaluation was funded by a Network Grant of the Royal Society of Edinburgh, Scotland’s National Academy. Scottish Charity No. SC000470.



Civic Epistemologies (2015). “Roadmad for Citizen Science”. Available also at

Clari, M. (2012). “Curating the Scottish built heritage record: Canmore as an open digital archive” International Journal of Heritage in the Digital Era, December 1(4), 535-566.

Economou, M. (2014). “Reinventing the Academic Museum: Studying the Digital Transformations at Glasgow’s University Museums.” In EPC 2014 Horizon2020 and Creative Europe vs Digital Heritage: A European Projects Crossover, Florence, Italy, 18 Feb 2014, 7-11.

Economou, M. (2016a). “The Scottish Network on Digital Cultural Resources Evaluation,” D-Lib Magazine, In Brief vol. 22(7/8). Consulted January, 25, 2017

Economou, M. (2016b). “Evaluating Digital Resources in Cultural Heritage: Lesson from the ScotDigiCH Network.” In: 22nd International Conference on Virtual System and Multimedia (VSMM), Kuala Lumpur, Malaysia, 17-21 Oct 2016, 99-105. Consulted March, 25, 2017 (doi:10.1109/VSMM.2016.7863188)

Hamilton, S. Jones, R. & McKeague, P. (2014).“Old places, New ideas: new routes into Canmore, the National Inventory of Scotland”, Archaeology in the Digital Era, 291.

Macdonald, S., Arnold-Forster, K. and Curtis, N. (2013). “Impact and Engagement: University Museums for the 21st Century.” In University Museums Group and University Museums in Scotland. Also available

Ridge, M. (ed.), (2014). Crowdsourcing our cultural heritage. London: Ashgate.

Royan, B. (1999). “Cross-domain access to digitized cultural resources: The SCRAN project.” IFLA Journal, 25(2), 101-103.

Tanner, S. (2012). “Measuring the Impact of Digital Resources: The Balanced Value Impact Model”, King’s College London, October 2012. Available at:

University Museums in Scotland (UMIS) (2004). Opening doors to learning. University Museums for 21st century Scotland. Also available


Cite as:
. "Use and Impact of Digital in Cultural Heritage: Insights from the Scottish Network of Digital Cultural Resources Evaluation." MW17: MW 2017. Published February 1, 2017. Consulted .

Leave a Reply