Source page 1

This blue sky paper envisions a novel system which promotes emotional closeness through storytelling. Family members, who may be separated, collaboratively build a spatial hypertext of images and text fragments to express and structure their thoughts and memories. The system observes their reactions as well as their media while they work. Live recommendations prompt users in their thinking and storytelling. Family stories are thus collaboratively adapted to more tightly connect the thoughts and emotions of their loved ones.

• Human-centered computing →Hypertext / hypermedia; Information visualization; Natural language interfaces; • Information systems →Collaborative and social computing systems and tools; Recommender systems.

emotional closeness, memory space, remembrances, collective memory, family memory, intergenerational, storytelling, long-distance relationships, hypertext

ACM Reference Format: Claus Atzenbeck, Mark Bernstein, and Sarah Diefenbach. 2022. Emotional Closeness by Means of Intelligent Thoughts and Memory Spaces . In Proceedings of the 33rd ACM Conference on Hypertext and Social Media (HT ’22), June 28-July 1, 2022, Barcelona, Spain. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3511095.3536363

Hypertext, recommender, and natural language systems all frequently target the domain of knowledge management. Such systems are used to build or share knowledge among cooperating persons, most often in the workplace. Here, we examine instead the collective sharing of personal experiences and stories within a family or among close friends. This is more than simply sharing knowledge: it includes subtle apprehension, taboos, and customs that may span generations and evoke strong emotions. Halbwachs [7] coined the

∗Corresponding author

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). HT ’22, June 28-July 1, 2022, Barcelona, Spain © 2022 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-9233-4/22/06. https://doi.org/10.1145/3511095.3536363

term “‘collective memory’ to describe the memory of a group that transcends the remembered experiences of its individual members” [12]. This memory is built upon communication among group members and “conveyed to future generations through family stories” [11].

In telling these intimate narratives, the roles of storyteller and listener change frequently. There is no clear border in between them, as “participants trying to learn and build their family memory were active seekers and creative participants” [11]. Further, Jones and Ackerman argue that the stories are shaped over generations, and that today’s stories will be taken over by family members not yet born [11]. Over time, people change their perspectives. For example, some aspects of a family story are not intended for young children’s ears. Those children may learn more of the stories when they grow older and eventually even change their perspectives on them over time.

Although the storytelling takes place often face to face, other channels may be used, too, including letters, phone calls, email, or chat clients. Some systems propose a shared display for synchronous interactions over distance [24], others used asynchronous communication via audio recording [10]. Researchers have also explored systems that increase the sense of closeness [8]. Building family memories is thus assisted by “multi-lifespan information systems mediating asynchronous, asymmetric, co-constructive and socially weighted information sharing interactions” [11].

In this paper we propose a system that pursues communication of emotionally-close people over distance. (Synchronous communication would also be possible, for example, via shared screens, but this is not the main target of this paper.) Furthermore, it allows multiple perspectives of “memory artifacts” [11] and spontaneouslychanging roles of storytellers and listeners. That is, a user may take both roles at the same time: telling (via structuring and speaking) and being told (via machine recommendations) takes place synchronously.

The system collects information from the users’ playfully-created visual structures of thoughts and remembrance artifacts in a 2D space. As they work, the system also records their verbal comments, analyzes and segments them, and associates comments with the specific digital artifacts. Emotions observed during the interaction, derived from speech, facial expression, gesture, or physiological measurements [5] may suggest prompts or oracles to propose additions to the memory space.

Overall, the system enables users to create their own perspective by structuring artifacts and “telling their stories”. The result of the machine’s analysis is an emerging knowledge graph, referring to users’ tacit knowledge, memories, or perspectives in association

Source page 2

The spatial hypertext provides an interesting narrative challenge and opportunity. Though we sometimes imagine that stories are told from beginning to end, or that they follow some sort of chronological sequence, real stories tend to be far more complex [6, 17, 25]. A story might naturally begin with its motivation: “How did you meet dad?” The date of that meeting is in our past, and the story that leads up to it is even more distant. That meeting, in turn, might lead very naturally to narrative that takes place after the meeting (“At first, we did not get along...”) or allude to things that happened much later (“Even then, he was crazy for Tchaikovsky.”) This anachronic complexification is even more pronounced in memoir, which almost inevitably concerns the elder’s confrontation with their remembered younger self [14], and also, in the Memory Space, their dialogue with a child or grandchild.

The open-ended two-dimensional plane provides a more flexible space for the discovery of emergent structure than does a simple list or timeline. Crucially, we can place an image or a note where it might provisionally belong without premature commitment to its position in some sequence [21]. Unconstrained by a predetermined formalism, we are free to create a multiplicity of overlapping sequences that compose a braided narrative of lived experience.

Oracles and Improv

As already noted, a spatial parser can interpret what the participants actually did and adjust the result to align that action with the participant’s intent. If several similar items are aligned in a row, and a new item is placed nearby but slightly misaligned, we might propose to nudge it into place [20]. If we have added a snapshot of a wartime family dinner in a subway tunnel, and the computer observes that the color balance of the image reflects aging pigments, it might offer to rebalance the hues or suggest converting to gray scale or duotone. Automatic facial recognition can be pernicious in a surveillance state but may be a blessing when reviewing old family photographs: yes, indeed, that is your cousin Isabel (Irving’s oldest girl).

The machine has little insight into the story that is in the process of becoming [13]. We don’t know it ourselves. Even so, computational characters [2] might usefully pose questions to open overlooked lines of discussion. One, whom we might call the Wise Child, might be inclined to ask for more specific and detailed information: “You mentioned a wartime dinner in a subway station: which stop was it?” Another might urge us to pursue narrative branches: “Whatever happened to cousin Isabel after the War?” These charac-

ters are a chorus of “oracles”—computational writing prompts. We may give those that seem interesting greater scope [3] and mute those we dislike—though the improvisatory-dramatic ethos of “yes, and”—accepting the prompt and, if necessary, redirecting it—might be a productive approach [19].

These oracles will not always be correct, but reflecting on their suggestions may lead the narrative in profitable new directions.

Source page 3

Indeed, opening up the story might best be accomplished by an amicable and interested character who is curious but not very bright. Montage and collage use the juxtaposition of two images to create a new idea that is contained in neither the first or the second image [4, 16]; we can easily dismiss erroneous suggestions, but by reflection we might gain new insights.

In this blue sky paper we presented the use of intelligent, NLP and recommender enabled spatial hypertext systems for increasing emotional closeness of close people, who are separated by time or space, such as family members living in different cities or time zones. In particular we discussed related narrative and psychological aspects. The novelty of such system is its combination of (i) an intelligent, cognitive map-based user interface; (ii) its ability to analyze users’ spoken comments, thoughts, or stories; (iii) detecting users’ emotions while interacting with the system; (iv) its ability to create knowledge graphs based on multiple users’ inputs, including spatially structured objects, verbally expressed comments, and emotions; and (v) recommender functionality for augmenting users’ narrative contexts with suggestions derived from the system’s knowledge bases. The system broadens users’ thinking by confronting them with thoughts, ideas, or emotions from close people. By this, it increases emotional closeness between humans even if separated by time, space, or generation.

From a psychological perspective, the system touches upon several facets of relatedness, including awareness, memory, and joint action. These have been discussed as prominent strategies of technologymediated relatedness in the Human-Computer Interaction (HCI) literature [9].

Awareness refers to creating a sense of contact by sharing information without explicit conversation. If Maria is confronted with a thought that Otto has added, this creates awareness of Otto’s thinking in an unobtrusive way, without expectation of a direct reply. In the HCI and design literature, a prominent design approach closely related to awareness is the concept of “ambient design”, which aims at peripheral, implicit experience (e.g., [26]). In contrast to explicit, conversational technologies such as the telephone, ambient technologies are designed to fit into daily routines and activities without causing much disruption [9]. In daily life, relatedness through awareness, appears in many forms. A simple example given by Hassenzahl et al. is hearing the children at play in the back yard as you work at your desk in your study [9]. There is no explicit communication in either direction, but being aware of what your loved ones are doing strengthens feelings of relatedness.

Memory as a relatedness strategy refers to keeping a record of past activities and special moments of a relationship. As discussed in [9], memorabilia often to mediate relatedness in moments of separation (e.g., wedding bands), and couples and family members tend to collect souvenirs, pictures and other things that evoke their shared history [18, 23]. In this regard, the system provides a new space for memories, one that transcends photo albums or social media, and in which family members together may reawaken memories.

Oral history is hard, and can be even more difficult when it engages the mystic chords of familial affection. To explore unfamiliar memories, we might need to examine things that are uncomfortable. In some cultural and societal contexts, it may be difficult for young people to question older relatives. For these reasons, a Memory Space is an artifact as well as a collaborative activity. Neither an interrogation nor an inquisition, it offers a chance to work socially as we arrange and explain images and narratives. Since the system highlights the connections between Maria’s and Otto’s content, they are reminded of how their thoughts and stories are connected, strengthening their sense of relatedness.

The system provides a new space for joint action and shared activities, as a central element to strengthen and reinforce relationships [27]. Though the system is not a space for explicit conversation, it can be a trigger for this. Family members can “play” with it together, experiment with the system, arranging and re-arranging different types of content, reflecting and talking about the personal motivations and meaningfulness of specific content, or they may combine the system with conventional conversation.

[1] Claus Atzenbeck and Daniel Roßner. 2020. Thoughts Reflection Machine. In

Proceedings of the 31st ACM Conference on Hypertext and Social Media (HT ’20). ACM Press. https://doi.org/10.1145/3372923.3404837 [2] Mark Bernstein. 2020. Bad Character: Who Do We Want Our Hypertexts to Be?.

In Proceedings of the 31st ACM Conference on Hypertext and Social Media. ACM Press, 221—224. https://doi.org/10.1145/3372923.3404777 [3] Mark Bernstein. 2020. In Spite Of Ourselves: Our Duty To Our Slaves Of Steel. In

Proceedings of the 3rd International Conference on Web Studies (WS.3 2020). ACM Press, 4—8. https://doi.org/10.1145/3423958.3423961 [4] Sergei Eisenstein. 1942. The film sense. Harcourt, Brace and Company. [5] Jose Maria Garcia-Garcia, Victor M. R. Penichet, and Maria D. Lozano. 2017.

Emotion Detection: A Technology Review. In Proceedings of the XVIII International Conference on Human Computer Interaction. ACM Press. https://doi.org/10.1145/ 3123818.3123852 [6] Gérard Genette. 1983. Narrative Discourse: An Essay in Method. Cornell University

Press. [7] Maurice Halbwachs. 1992. On Collective Memory. University of Chicago Press. [8] Marc Hassenzahl, Sarah Diefenbach, Kai Eckoldt, Stephanie Heidecker, Uwe Hill-

mann, and Matthias Laschke. 2010. Technologie, die verbindet? Die bedürfniszentrierte Gestaltung von Kommunikationstechnologien für Liebende (und andere, die sich mögen). Usability Professionals (2010). https://dl.gi.de/bitstream/handle/ 20.500.12116/6078/Hassenzahletal2010.pdf [9] Marc Hassenzahl, Stephanie Heidecker, Kai Eckoldt, Sarah Diefenbach, and Uwe

Hillmann. 2012. All You Need is Love: Current Strategies of Mediating Intimate Relationships through Technology. ACM Transactions on Computer-Human Interaction 19, 4 (2012), 1–19. https://doi.org/10.1145/2395131.2395137 [10] Yasamin Heshmat, Carman Neustaedter, Kyle McCaffrey, William Odom, Ron

Wakkary, and Zikun Yang. 2020. FamilyStories: Asynchronous Audio Storytelling for Family Members Across Time Zones. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM Press, 1—14. https://doi.org/10. 1145/3313831.3376486 [11] Jasmine Jones and Mark S. Ackerman. 2018. Co-Constructing Family Memory:

Understanding the Intergenerational Practices of Passing on Family Stories. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM Press, 1–13. https://doi.org/10.1145/3173574.3173998 [12] Jasmine Jones and Mark S. Ackerman. 2020. Learning from Family Mysteries:

Accounting for Untold Stories in Family Memory Practices. Proceedings of the ACM Human-Computer Interaction 4, CSCW3 (2020), 221:1–221:22. https://doi. org/10.1145/3432920 [13] Michael Joyce. 1988. Siren Shapes: Exploratory and Constructive Hypertext.

Academic Computing (11 1988), 11ff. [14] William Manchester. 1980. Goodbye, Darkness: A Memoir of the Pacific War (3

ed.). Little Brown. [15] Catherine C. Marshall and Frank M. Shipman. 1995. Spatial hypertext: designing

for change. Commun. ACM 38, 8 (8 1995), 88–97. http://doi.acm.org/10.1145/ 208344.208350 [16] Scott McCloud. 1993. Understanding Comics. Kitchen Sink Press. [17] John McPhee. 2013. Structure: Beyond the Picnic-Table Crisis. The New Yorker (1

2013), 46–55.

Source page 4