Measuring the Emotional Experience of Visitors to Dark Sites

While there is clearly a need for more data on visitor experience to museums, including dark sites, there is a need for more effective and efficient data collection methods. Data collection should be done in such a way that it does not hamper the visitor experience while being as informative as possible. Additionally, researchers strive for a combination of both subjective and objective measures to obtain a more rich and complete data set. In this post I will describe a proposed method of collecting both objective and subjective data regarding visitors' emotional experience developed by Doreen and I.

First I will describe what type of emotion data we will be able to collect. As I mentioned earlier we will want to collect both subjective and objective measures. Subjective measures are explicit measures, commonly in the form of self report measures. Self report emotion measures simply ask the individual to indicate how they are presently feeling (state emotion) or how they generally feel (trait emotion). There are several emotion self report measures which can easily be completed by the visitor. We can also create a new self report questionnaire for our purposes using items from existing self report measures. That way we can make sure we collect data for specific items of interest while limiting the length of the questionnaire, increasing the likelihood visitors complete the questionnaire.

Perhaps more interesting is how we can collect objective emotion data. This is where my research and content from my previous blog posts come in. If visiting dark sites have an emotional effect on visitors, it may very well temporarily bias their cognitive functions, such as their memory, interpretations, and perceptions. I am particularly interested in perceptual biases, where my research investigates biases associated with the perception of neutral or ambiguous faces. If dark sites illicit powerful emotional experiences, we may be able to assess this by determining whether visitors demonstrate any negative cognitive biases following their visit. To assess this we would use versions of the emotion recognition training task. Visitors would only complete what is traditionally the first part of the task: visitors would be presented a range of images of an individual portraying emotional expressions ranging from happy to a negative emotion (anger, disgust, fearful, sadness) including images portraying neutral or ambiguous expressions. Based on how many of these images visitors rate as a negative emotion rather than happy, we could determine whether they are biased towards more negative perceptions, and identify the specific negative emotion associated with the bias. This is an objective measure given participants are not explicitly being told what the task is assessing.

A negative perceptual bias where more faces are perceived as sad rather than happy.

Another way we can collect objective emotion data would be using the emotion recognition software described in my last post. If it was possible to set up a live video feed by placing video cameras at various locations throughout the sites, we would be able to collect emotional data using software such as FaceReader or CERT would allow us to collect emotion data in real time. In fact, the software could do this while maintaining the anonymity of the visitor, by simply recording the response (or emotional data) without recording or storing the image of the visitor. That means that using the live video in conjunction with the software, we could collect information about emotional expressions portrayed by visitors, giving us insight into their emotional experience at that very moment. Again this is especially effective given it would give us data on their objective emotional experience rather than subjective, self report data. If a live video feed is not an option, video can be recorded and then sent to Emotient. However doing so would require additional consent to be obtained from the visitors given their image would be stored. As such, a live video feed would be more effective.

Finally, it is important to consider the delivery of these measures. Data collection is most effective when it is done efficiently and with little time commitment to the visitor. Additionally we must remember that for emotion data to be most accurate it should be collected before, during, or immediately after a visit, preferably before the visitor leaves the site. In order to accomplish this in order to collect the emotional data described above, we turn to the audio guide.

Audio guides are available at most museums and many dark sites and offer us with a very efficient and effective way of collecting data. While traditional audio guides were generally limited to simple number input and the presentation of audio recordings, audio guides have seen important technological advances over the last couple of decades.

Newer audio guides are now able to show full colour images and text, and allow visitors to input more varied information. The physical hardware for audioguides has substantially changed as well, with some museums now using Apple iPhones to deliver the audio guide program to their visitors. We believe using an audio guide with features similar to a smart phone, such as full colour images, extensive input options, as well as front facing cameras, would be the most effective way to collect emotional data. Visitors could complete self report measures and questionnaires presented by the device's screen and quickly input their response using the touchscreen functionality. Given that many popular smart phones now have front facing cameras, these could be used to capture the live video necessary for objective emotion data collection mentioned earlier. Attention can be directed to the screen when displaying colour images. At those moments when visitors are reacting to what they are viewing on the screen, the font-facing camera can be used to deliver live video to emotion recognition software and their expression data can be recorded. Finally, the emotion recognition training task has already been adapted for smartphone use and can run on the same smartphone as the audio guide before and after the start and end of the visit. In addition to these emotion measures, we will collect sociodemographic data using a visitor survey developed by Doreen as part of her PhD.

Therefore, using a smart phone audio guide as the data collection method, we would be able to obtain rich data regarding the emotional experience of visitors to museums and dark sites. Given the data collection would be integrated into their audio guides, data would be collected easily and efficiently, not requiring the visitor to use an external data collection device and would be a practical way for them to complete questionnaires and surveys. Given that many museums and sites are already using such devices as audio guides, developing the software to combine the existing audio guide content to include these measures would not require too much further effort and would be extremely effective for data collection. This offers museums and dark sites a novel and effective way of collecting emotional and sociodemographic data about their visitors in an unprecedented way and allow us to better understand the emotional experiences related to visiting dark sites in the future.

Visitor Research at Museums and Heritage Sites

As mentioned in my last post, there are a lot of questions regarding visits to "difficult" heritage sites and museums. First and foremast, how do visitors engage with an exhibition?

Visitor research at museums and heritage is often concentrated around evaluation, e.g. the success of an exhibition and market research. The most extensive research regarding visitor behaviour has been conducted by the market research company Morris Hargreaves McIntyre. They divided visitors into eight cultural segments based on their motivation. Whilst these segmentations are useful in terms of future marketing, they do not highlight how visitors engage with an exhibition. I also wonder whether these segments would apply for "painful" sites?

Exhibition evaluations tend to focus on learning outcomes which is measured through visitor survey research, focus group research and visitor observation. I remember attending a training course on visitor research at the British Museum and they discovered that visitors tend to spend a maximum of 60 seconds. So do visitors to museums actually engage with exhibits if they don't read labels?

All the previously mentioned research methods have their place in museums and heritage sites however, they only provide us with a brief snapshot of people's experiences to these sites. We often don't know whether people's experiences change throughout the visit and if so, how do they change? We also don't know how visitors engage emotionally with exhibitions. This is particular relevant to "painful sites" or "memory museums" as Silke Arnold-de Simini (2013) calls them in her book "Mediating memory in the museum". Memory museums are characterised by being object poor as ultimately perpetrator did not leave much traces. So if there aren't many exhibits, how do we research the visitor experience?

In Germany, the main aim of the memorial sites is the message "Never again" hence, the focus is very strongly on learning outcomes. Yet, we do not know whether visitors really do engage with this message. Measuring emotions before, throughout and after the visit would provide us with a real insight into how visitors experience traumatic sites. We would know which emotions visitors experience throughout their visit. Are they overwhelmed and therefore they start to disengage or do they feel indifferent? Which influence do architectural clues have on the visitor experience (the Jewish Museum in Berlin created a Holocaust exhibition with a "dead end")? Are these clues necessary?

All these questions remain currently unanswered and I therefore think, "researching" emotions in museums would transform our knowledge about visitors to heritage sites and museums.

Facial Expression Recognition Software

Following my last post, I will now describe a selection of the various facial expression recognition software and programs currently available.

FaceReader, developed by Noldus, is described as "the premier professional software for automatic analysis of facial expressions". The software works by automatically analyzing visual input for the 6 basic emotions I mentioned in my last posts (Anger, Disgust, Fear, Happiness, Sadness, & Surprise) as well as neutral and contempt. In addition it also assesses gaze direction, head orientation, and person characteristics. It also offers analysis of Action Units. The software was trained using more than 10,000 manually annotated images and offers accurate modeling of the face by describing 500 key points. It is presently in its 6th iteration and claims to have been used worldwide at more than 300 universities. See it in action in the above video.

 The Computer Expression Recognition Toolbox , or CERT, is described as "an end-to-end for fully automated facial expression recognition that operates in real-time". It was developed at the University of California, San Diego, stemming from a collaboration between researchers Paul Ekman (mentioned in my previous post) and Terrence Sejnowski. The software works by automatically detecting frontal faces in the video stream and codes each frame to 40 continuous dimensions, including the 6 basic expressions, contempt, head pose, and 30 facial Action Units. In the above video, they show how this system is being used as an intervention for children with autism. Possible applications include automatic discrimination of fake or posed expressions, detection of driver drowsiness, and adaptive tutoring systems.

Emotient is described an on-demand web service where users can upload video from focus groups, online panels and other sources and quickly receive reports and data about the emotional state of individuals in the video, as well as information about their attention and engagement. It seems to be largely marketed for commercial use to provide this information to companies when customers are experiencing their marketing, products, websites, or retail location. While this does not work with live video, it may be an interesting alternative to when live video is not possible or sending recorded video is easier or preferable.

SHORE is an object and face recognition software developed for the Google Glass. Adapted to recognize facial expressions of emotion, the technology has been trained by accessing a database of more than 10,000 annotated faces, similar to FaceReader. The developers, researchers from the Fraunhofer Institute in Germany, claim that in combination with the structure-based features and learning algorithms, they can train "models" that boast extremely high recognition rates. They cite a 92% facial recognition detection rate, as well as highly accurate gender detection and age estimation.  However the technology is still only currently limited ro the detection of four facial expressions: Angry, Happy, Sad, and Surprised.

And now, something for social media! Pocket Avatars, an app developed through Intel Labs,captures emotions in real time using facial-tracking technology to then send to your friends in the form of an animated avatar. Your friend then receives a video message where the avatar relays the emotion by mimicking your posed expression. These messages can run up to 15 seconds in length and are sent the same way as a Facebook or WhatsApp message. The app uses your smartphone's front facing camera, and allows you to also send a voice message that the avatar's movements sync to accordingly.

Based on this post it is evident there are quite a variety of facial expression recognition software available, ranging in price, functionality, and use. Some of these are free or offer free trials, so go ahead and try them yourself and let us know what you think!

Classifying Facial Expressions of Emotion

There is a large body of research regarding classifying facial expressions of emotion. I will be presenting one popular theory relevant to our collaboration here.

The Facial Action Coding System (also known as FACS) is a system used to classify human facial movements based on a system developed by Swedish anatomist Carl-Herman Hjortsjo and later adopted by Paul Ekman and Wallace Friesen, pioneers in the research of facial expressions of emotion. It works by coding the movements of individual face muscles and as you may have guessed, has been used to systematically categorize physical, facial expressions of emotion based on these specific muscle movements. It  first deconstructs the facial expressions into muscle movements, and then further encodes them as Action Units (AUs), representing the contraction or relaxation of one or more muscles.

Therefore, FACS can be used by man and machine alike to correctly label expressions. Given that FACS categorizes facial muscle movement, its use can extent beyond humans to several non-human primates, including chimpanzees, rhesus macaques, gibbons, and orangutans, with a recent adaption for dogs as well.

Commonly considered the 6 basic emotions, Happiness, Sadness, Surprise, Fear, Anger, and Disgust are represented by the following AUs:

Happiness	6+12
Sadness	1+4+15
Surprise	1+2+5B+26
Fear	1+2+4+5+7+20+26
Anger	4+5+7+23
Disgust	9+15+16

However, using AU and the FACS system, researchers from Ohio State University have identified what they consider to be 21 distinct emotion categories derived from compound facial expressions of emotion, each with their own unique AU combination (Du, Tao, & Martinez, 2014).

Given that emotional expressions can be broken down into action units that can be observed and measured,  this opens the door for the development of computer programs that can detect expressions in real time using live video or image capture. I will describe some of these programs in my next post. 

Some content from this post has been taken from the FACS Wikipedia page. Images from this post acquired using Google Images.

Tourism to painful places

It is evident that visitor numbers to “painful/traumatic” sites are rising but what is behind this rise and how do people feel during and after a visit? I thought I’ll share two of my own experiences.

In 2011 I went on a holiday with my partner to Austria cycling along the Danube. When cycling along the Danube you inevitably have to cycle through Mauthausen, a pretty Austrian village but also the site of Austria’s largest former concentration camp. Due to my PhD (and my personal interest), my partner (who is English) decided to visit the memorial. We initially visited the variety of memorials established within the camp boundary when I suddenly realised that my partner was in tears after reading one of the inscriptions. He was so upset that he left the site. I continued visiting the rest of the memorial and although I was moved, I was not emotionally involved. Having grown up in Germany, I have visited memorial sites before and I am very familiar with the atrocities in these sites – I wonder whether prior knowledge makes a difference to visitors?

In 2014, I had to opportunity to visit the former KGB building in Riga. It was the first time since 1991 that the building was open to the public (as part of Riga’s European Capital of Culture celebrations). After the KGB had left, the building remained empty and no alterations to the building have been made since then. Hence, there was hardly any “tourism” infrastructure (e.g. in the form of exhibitions), the former KGB prison cells had remained in their “raw” state. For me, this visit was incredibly moving (much more so than Mauthausen). It reminded me of my time in East Germany – although I was very young I do remember my uncle being imprisoned due to political reasons. Are visits to sites which are still in living memory more emotionally moving? Or do visitors have to have a personal connection in order to fully understand the site? If so, how do we replicate this experience if there is no longer a personal connection? 

According to the Dark Tourism Research Centre at the University of Central Lancashire, cemeteries are also dark tourism sites. The most well-known site in the UK is Highgate Cemetery in London or the Necropolis in Glasgow. Whilst studying for my MA in Heritage Management I conducted visitor research at Arnos Vale Cemetery in Bristol. The results did not confirm that the site is a Dark Tourism sites. Visitors who attended the guided tours had a strong desire to find out more about the social history of the site and wanted to enjoy the wildlife in an otherwise very busy part of Bristol. Is it therefore appropriate to call these sites dark tourism sites? And more importantly, is death dark? I hope that this post demonstrates the complexity of the “Dark Tourism” concept and the importance of visitor research to gain an understanding of this complexity.

My research

My research in Dark Tourism commenced during my final year at the German university. As part of my undergraduate degree I worked for six months at Colditz Castle in Germany (famously known for its past as a Prisoner of War camp). During this time, I realised how difficult it can be to display sensitive issues in museums and how little is known about visitor’s experience at these sites.

Visitors to “difficult” sites have risen sharply, Auschwitz now has 1.5 million visitors (http://www.thenews.pl/1/9/Artykul/192671,Auschwitz-draws-record-number-of-visitors).  In order to explain this phenomenon, a research group has established itself at the University of Central Lancashire calling it Dark Tourism (http://dark-tourism.org.uk/). 

I am very concerned about the term Dark Tourism as it implies that a visit to these sites is very gloomy and visitors have a fascination with the macabre and/or death (Preece and Price 2005). Furthermore, several categories for Dark Tourism were developed which range from amusement parks to concentration camps.

I think, the current concept of Dark Tourism is too simplified. It does not consider the different cultural forces which create these places in the first instance. Commemorative heritage sites are often established by the local community or a group who had to endure suffering at the site; the principal aim is usually to maintain the memory and has nothing to do with attracting visitors who want to engage with death. Visitors to these sites often feel a strong sense of obligation (“I owe it to the people who died here”) and can feel strong emotions. A recent study at the former political prison in Bautzen (East Germany) has revealed that visitors have a strong desire to see the real place, wanting to see for themselves where atrocities have happened (Pampel 2007). My aim for my research project is to gain a deeper understanding of the visitors to these sites, how they engage emotionally with the site and whether the site has a deeper long-term impact.

In June last year, I had the opportunity to visit the former KGB building in Riga/Latvia. It was the first since Latvia’s independence that the building was open to the public. I was very moved by the experience (and so was everybody else) and having talked to some visitors, it was apparent that they did not visit the site for macabre reasons, it was predominantly for educational purposes and/or discovering family history.

The article with the following headline “Nazi relics could be ‘dark tourism’ sites, says Glasgow academic” in The Times, Scotland http://www.thetimes.co.uk/tto/news/uk/scotland/article4411763.ece. Professor John Lennon will explain to tourism experts in Germany how to create “Dark Tourism packages” which fascinates visitors. Is this appropriate? And if yes, are there any sites which cannot be included in these packages?