EFMHuman Computer InteractionProjectsSOCIETIESUser Trials

SOCIETIES Final Prototype “In The Wild” Student Trial at Heriot Watt University, Edinburgh

By 2nd December 2013 No Comments

by Edel Jennings (EFM/TSSG)

From its conception, the EU FP7 SOCIETIES project has been focused on engaging with ‘real’ users, through participatory user-centred-design processes, to design, develop and evaluate an open scalable service architecture and platform for communities using social and pervasive computing. This system, which has now been developed and deployed in user trials, enables proactive smart space behavior. It has been designed to facilitate advanced community activities, such as the discovery, connection and organization of relevant people, resources and things across real and digital spaces. Now, in the closing stages of the project, the final prototype trials have been or are taking place. In the case of the final Student trial, which is described in this post, SOCIETIES researchers had a rare opportunity to evaluate people’s experiences, and interactions over a longer time period, in an immersive social and ubiquitous SOCIETIES system, which was deployed in Heriot Watt University in Edinburgh, with up to twenty students participating for six weeks.

Tree of Innovations from SOCIETIES

Diagram 1.0: Tree of Innovations from SOCIETIES


The innovative features afforded by the SOCIETIES platform, (and shown in the innovation tree diagram 1.0) have been leveraged, to develop third party services in cooperation with people from three distinct groups: a Student group, an Enterprise group and a Disaster Management group. These three groups have been working with SOCIETIES to envision scenarios and test prototypes throughout initial user research activities and the three phases of user trials in the project’s lifecycle.

Data Gathering & Trial Activities

Edel Jennings from TSSG’s Experimental Facilities Management group visited SOCIETIES partners at Heriot-Watt University (HWU), in late November, to assist the HWU trial management team, in conducting interviews with the trial participants. Each participant took part in an hour-long in-depth interview, which was recorded and videoed. This student trial has been the most ambitious trial in the SOCIETIES project and it may be expected that the rich and varied data captured, through mobile logging, interviews, questionnaires and participatory activities will provide a valuable seam for further analyses.

Trial Description

The Student Trial has had up to twenty students actively participating in an immersive ‘in the wild’ trial, for the six weeks. Each student was provided with an android mobile device and an RFID tag, which they agreed to use for the duration of the trial. The trial also featured smart screens in situ at one of the university’s learning zone, along with MS kinect devices, and geofences. The participants were able to install their personal SIM cards to the devices, encouraged to download applications and to generally employ the devices both on and off campus, for personal use, which several chose to do. During the trial, each student used up to six different SOCIETIES third party services, in conjunction with the SOCIETIES mobile App and Webapp. The students could engage freely with each other, via the system’s communities, and on occasions also opt to take part in planned group participatory activities, which were facilitated by the trial managers, which gave them a rich and varied immersive experience of the SOCIETIES system. Usage of the system varied considerably amongst the participants, where a few individuals emerged as lead adopters.

There were many challenges in managing such a long running trial for pervasive systems, both from  engineering and evaluation perspectives.  As Andy Crabtree et al have explained, there is a need for new tropes for “in the wild” trials as in their experience while trials are useful; they are complex to deploy and end of development field trial evaluations may not indicate the technologies readiness for wider appropriation. “The model stands in need of amendment and we would also suggest that, as part of this,the emphasis on evaluation be replaced with an emphasison a continuous community-oriented process of reflection,elaboration and modification shaping the co-realisation of innovative systems.” [1]

Users had to accept adaptation to the prototype quality services, which do not have the final polished appearance or smooth running of commercial apps,  during the “in the wild” trial situation, due to deployment and usability issues. However, the student participantss quickly became accustomed to the prototype quality, and were able for the most part to achieve their goals using the basic mobile and web interfaces provided. The gestural interactions using the kinect with the public screens were novel and not intuitive, which appeared to have polarized the participants. Some participants liked the novel interactions very much, whereas others were frustrated or confused. One mentioned that the gestures were not required for the task at hand (selecting screens for personalized Television programmes or playing quiz games) and therefore superfluous, whereas others envisioned scenarios where gestural interactions and screen selection based on proximity would be useful, as in the example provided by one participant of a person cleaning several rooms in the home, carrying equipment from room to room, accessing their favourite soaps on the various screens throughout the home as they go. Similarly there are evident idiosyncrasies in how the individual services were perceived from person to person: the Collaborative Quiz, for example, was very popular with some, but for others it was their least preferred service. GeoFencing, which allowed students to experiment with setting up digital boundaries in certain regions, was highly regarded by several students, who experimented with using it in different situations, but was found to be disappointing by others. Factors regarding trust, privacy and automation, were given great consideration, in the interviews, as in the systems’ development, with extensive privacy policies making users conscious with each sharing interaction, while often they were accepted without reading, their presence led to the participants feeling protected and safe using the system.  No one felt that they had modified their behavior because they were using the system.

The SOCIETIES Services deployed were –

  1. Askfree (HWU) – A service allowing students to post questions or comments during lectures.
  2. CoBrowse (NEC) – A service enabling two or more remotely located users to simultaneously interact to co-browers and communicate using voice and audio.
  3. Collaborative Quiz (HWU) – This was a quiz game, situated in a public space, where students interacted with a smart screen using a gestural interface.
  4. GeoFencing (NEC) – This is a service, originally designed for disaster management situations, which enables people to draw digital geofences and interact with communities around the designated spaces.
  5. MyTV (HWU) – This service personalised auto selected screens and television channels in a public space based on the learned preferences of the student participant.
  6. SOCIETIES Calendar (PTIN) – This supported community events, notifications, and management.

In addition to the free use of the system and services by trial participants, Heriot-Watt University also organised up to four events for participatory activities, where attendees were invited to simultaneously play games and experiment with the services. These events provided a ludic element, and  were designed to enable participants to interact and experience social aspects of using such a system across actual and virtual situations.

Participatory Events organised –

  1. Quiz (CollabQuiz) – Two teams competed against each other.
  2. Treasure Hunt (GeoFencing) – A treasure game was organised to facilitate several users to use geofences at the same time.
  3. Race Game (CoBrowse) – A car racing game was designed to enable students use co-browse.
  4. Word Game (AskFree) – This game showcased the features of this community question and comment service designer to be used during presentations or lecturers.

Individual Participant Interviews

The final participant interviews, consisted of over forty questions, and the responses from the participants were open-ended and varied. The interviews have been recorded by video, where participants permission was granted, which will provide persistent records allowing for deeper analysis in the coming weeks and months. Using video facilitates data recording and frees up the interviewer to concentrate on communicating with the participants. It also allows for easier cross analysis of participant responses to particular questions. With up to an hour given to each participants interview, HWU have gathered a large amount of detailed data about users’ reflections on their engagement with the trial, SOCIETIES in general, each of the featured services, and other platform features including:

  1. Android App
  2. Web App
  3. RFID Tags
  4. Community management
  5. Activity feeds
  6. Personalisation
  7. Location awareness
  8. Privacy protection
  9. Data Sharing
  10. Social Networks
  11. Context Awareness
  12. Other Platform components

In addition to these in-depth interviews, Heriot Watt have logs of the systems usage by the participants over the six weeks, and questionnaire results intermittently conducted during the trials, which will provide further data for co-relation and contrast with the interview data.

The participants were clearly pleased with the level of support they received from the beginning to the end of the trial, which is indicative of the dedication and professionalism of the Heriot Watt trial team. Several said they had fun and enjoyed the experience overall. They were forthcoming in their feedback about the system and services. Several had interesting ideas about alternate uses for the system and services.

The nascent and novel nature of the system and third party services meant that deployment was restricted to a limited number of participants, which somewhat curtailed natural social community formation and activity, denying the possibility of a genuine network effect.  Those participants encountered many usability and platform issues, which interrupted users from free flow engagement with the system, and challenged the trial managers. Of course, the situation where the trial took place in a university, and the system administrators were known to participants and available to assist, may be said to be introduce known biases, or demand characteristics in the interviews (where some participants may be inclined somewhat to do or say what they feel expected of them), but “clean room” experiments of pervasive systems is not regarded as useful as the systems as so complex [2]. Despite these many difficulties, the co-operative ambience of the trial situation and the supportive relationships between the participants and the trial managers, meant that through hard work and persistence, SOCIETIES has now been successfully deployed and evaluated in a real world university environment.

The approaches and methods used will be documented in detail, in the final report.   The video interviews will be logged, and analysed using an interpretive qualitative approach. These results will be compiled and contrasted with the results from the mobile logging, which detail usages, connections with friends and communities, and services used, along with the data from the trial’s weekly questionnaires, and presented in SOCIETIES Final Prototype Trial results deliverable, along with the results from SOCIETIES other trial sites. The description of the process of running this trial and the results from this trial situation will we believe be of real value and interest to social and pervasive developers both in SOCIETIES and in the wider research community.


  1.  Doing innovation in the wild, CHItaly, p. 25, ACM, 2013.  ISBN: 978-1-4503-2061-0 doi:10.1145/2499149.2499150 by Andy Crabtree and Alan Chamberlain and Mark Davies and Kevin Glover and Stuart Reeves and Tom Rodden and Peter Tolmie and Matt Jones
  2. Exiting the Cleanroom: On Ecological Validity and Ubiquitous Computing HumanComputer Interaction, Vol. 23, No. 1. (2008), pp. 47-99, doi:10.1080/07370020701851086 by Scott Carter, Jennifer Mankoff, Scott R. Klemmer, Tara Matthews