Publications

Dissertation

Schraffenberger, H.(2018). Arguably Augmented Reality: Relationships Between the Virtual and the Real. (Doctoral dissertation). Leiden University, The Netherlands.

Abstract: This thesis is about augmented reality (AR). AR is commonly considered a technology that integrates virtual images into a user’s view of the real world. Yet, this thesis is not about such technologies. We believe a technology-based notion of AR is incomplete. In this thesis, we challenge the technology-oriented view, provide new perspectives on AR and propose a different understanding. We argue that AR is characterized by the relationships between the virtual and the real and approach AR from a fundamental, experience-focused view. By doing so, we create an unusually broad and diverse image of what AR is, or arguably could be. We discuss the fundamental characteristics of AR and the many possible manifestations it can take and propose new, imaginative AR environments that have no counterpart in a purely physical world.

Conference Proceedings

 
Tiemessen, J., Schraffenberger, H., & Acar, G. T. (2023). The time is ticking: The effect of limited time discounts on consumers’ buying behavior and experience. In CHI Extended Abstracts (p. 277:1–277:11). ACM.
 
Botros, L., Brandon, M., Jacobs, B., Ostkamp, D., Schraffenberger, H.K., & Venema, M. (2023). Postguard: Towards easy and secure email communication In CHI Extended Abstracts (p. 232:1–232:6.). ACM.
 
Brandon, M., Schraffenberger, H., Sluis-Thiescheffer, W., van der Geest, T., Ostkamp, D., & Jacobs, B. (2022). Design Principles for Actual Security. In NordiCHI (Adjunct) (p. 41:1–41:6). ACM.
 
Terpstra, A., Graßl, P., & Schraffenberger, H. (2021). Think before you click: how reflective patterns contribute to privacy [position paper]. In Workshop “What Can CHI Do About Dark Patterns?” at the CHI Conference on Human Factors in Computing Systems (CHI’21).
 
Borgesius, F.J.Z., Schraffenberger, H., & van Bekkum, M. (2021). Insurance, algorithmic decision-making, and discrimination [workshop description]. In PUMAP (Adjunct Publication) (p. 333).
 
van den Hout, T., Schraffenberger, H., Krause, F., Bosse, T., & Léoné, F. (2019). RunTheLine: An infinite runner serious game to train comprehension of societally relevant large numbers [abstract]. In Proceedings of the 41st Annual Meeting of the Cognitive Science Society (CogSci 2019) (p. 3596).
Schraffenberger, H., Sande, Y. van de, Schaap, G., and Bosse, T. (2019). Investigating people’s attitudes towards AI with a smart photo booth [extended abstract] In BNAIC/BENELEARN 2019. CEUR Workshop Proceedings, 2491.
Schraffenberger, H., Sande, Y. van de, Schaap, G., and Bosse, T. (2019). Can you fool the AI? – Investigating people’s attitude towards AI with a smart photo booth. In Proceedings of the Seventh Conference on Computation, Communication, Aesthetics & X, xCoAx’19.
de Rooij, A., Schraffenberger, H., & Bontje, M. (2018, March). Augmented metacognition: Exploring pupil dilation sonification to elicit metacognitive awareness.In Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction (pp. 237-244). ACM.

Abstract: Metacognitive awareness enables people to make conscious decisions about their own cognitions, and adapt to meet task performance goals. Despite the role of metacognition in task performance, technologies that effectively augment metacognition are scarce. We explore a novel approach to augment metacognition based on making the eye’s pupil dilations, which associate with a variety of cognitions, audible via sonification in real-time. In this exploratory study, we investigated whether pupil dilation sonification can elicit metacognitive awareness. Our findings suggest that correlations between a variety of cognitions, e.g., attentional focus and depth of thinking, and sounds generated by the sonification can emerge spontaneously and by instruction. This justifies further research into the use of pupil dilation sonification as a means to augment metacognitive abilities.

Schraffenberger, H., & van der Heide, E. (2017). Reconsidering Registration: New Perspectives on Augmented Reality. In Interactivity, Game Creation, Design, Learning, and Innovation (pp. 172-183). Springer, Cham.

Abstract: Augmented reality (AR) projects typically involve interactive systems that align virtual objects with the real world. This process is called registration and can make it seem as if virtual objects existed in the otherwise real environment. Registration is widely accepted as a defining and necessary characteristic of augmented reality. In this paper, we reconsider the need for registration on two levels. First of all, we argue that the intended presence of virtual objects in real space can be achieved without registration by an interactive AR system. Secondly, we suggest that the perceived spatial presence of virtual content in real space is not necessary for AR in the first place. We illustrate both points with examples and propose a more encompassing view of AR that focuses on relationships between the virtual and the real rather than on registration.

de Rooij, A., van Dartel, M., Ruhl, A., Schraffenberger, H., van Melick, B., Bontje, M., Daams, M., & Witter, M (2017). Sensory Augmentation: Toward a Dialogue between the Arts and Sciences.. In Interactivity, Game Creation, Design, Learning, and Innovation (pp. 172-183). Springer, Cham.

Abstract: People sense the world by exploiting correlations between their physical actions and the changing sensory input that results from those actions. Interfaces that translate non-human sensor data to signals that are compatible with the human senses can therefore augment our abilities to make sense of the world. This insight has recently sparked an increase in projects that explore sensemaking and the creation of novel human experiences across scientific and artistic disci- plines. However, there currently exists no constructive dialogue between artists and scientists that conduct research on this topic. In this position paper, we iden- tify the theory and practice of sensory augmentation as a domain that could benefit from such a dialogue. We argue that artistic and scientific methods can complement each other within research on sensory augmentation and identify six thematic starting points for a dialogue between the arts and sciences. We conducted a case study to explore these conjectures, in which we instigated such a dialogue on a small scale. The case study revealed that the six themes we identified as relevant for a dialogue on sensory augmentation emerge rather spontaneously in such a dialogue and that such an exchange may facilitate progress on questions that are central to the theory and practice of sensory augmentation. Overall, this position paper contributes preliminary evidence for the potential of, and a starting point for, a dialogue between the arts and sciences that advances our understand- ing of sensory augmentation and the development of applications that involve it.

Schraffenberger, H. & van der Heide, E. (2016). Multimodal augmented reality: the norm rather than the exception.. In Proceedings of the 2016 workshop on Multimodal Virtual and Augmented Reality. ACM.

Abstract: Augmented reality (AR) is commonly seen as a technology that overlays virtual imagery onto a participant’s view of the world. In line with this, most AR research is focused on what we see. In this paper, we challenge this focus on vision and make a case for an experience-focused and modalities-encompassing understanding of AR. We argue that multi-modality in AR is the norm rather than the exception, as AR environments consist of both virtual content and our real, physical, multimodal world. We explore the role multi-modal and non-visual aspects of our physical reality can play when creating AR scenarios and the possibilities and challenges that emerge when approaching AR from a modalities-encompassing perspective.

Schraffenberger, H. & van der Heide, E. (2015). Sonically Tangible Objects. In A. Clifford, M. Carvalhais & M. Verdicchio (Eds.), xCoAx 2015: Proceedings of the Third Conference on Computation, Communication, Aesthetics and X (pp. 233–248). Glasgow, Scotland, UK.

Abstract: A unique power of virtual objects is that they do not have to look, feel or behave like real objects. With this in mind, we have developed a virtual cube that is part of our real, physical environment but, unlike real objects, is invisible and non-tactile. ‘Touching’ this virtual object triggers binaural sounds that appear to originate from the exact spot where it is touched. Our initial experimentation suggests that this sound-based approach can convey the presence of virtual objects in real space and result in almost-tactile experiences. In this paper, we discuss the concept behind, implementation of and our experience with the sonically tangible cube and place our research in the context of tangible interaction, perception and augmented reality.

Schraffenberger, H. & van der Heide, E. (2014). The Real in Augmented Reality. In M. Carvalhais & M. Verdicchio (Eds.), xCoAx 2014: Proceedings of the Second Conference on Computation, Communication, Aesthetics and X (pp. 64–74). Porto, Portugal.

Abstract: What is augmented in Augmented Reality (AR)? This fundamental question has received surprisingly little attention in AR research. In this paper, we review exist- ing views and show how little consensus there is on the topic. Subsequently, we approach the question from a theoretical and technology-independent perspective that focuses on the relationships between the virtual and the real. We consider both spatial as well as content-based augmentations and distinguish between augmented environments, augmented objects, augmented humans, augmented content and augmented perception. We dis- cuss our findings and suggest possible future directions, such as research into multimodal and crossmodal AR.

Gómez-Maureira, M. A., Teunisse, C., Schraffenberger, H., & Verbeek, F. (2014). Illuminating Shadows: Introducing Shadow Interaction in Spatial Augmented Reality. Creating the Difference, 11.

Abstract: In this paper we present a new mode of interaction in ‘Spatial Augmented Reality’ (SAR) setups, using shadows as interaction input as well as display area. We claim that the combination of shadow interaction and SAR offers a novel, enjoyable and interesting way of interacting with information in a physical manner. This is especially relevant for contexts such as museum exhibits, where digital information and physical objects relate to one another. The results of our usability experiment with a zebrafish model show that users enjoy the combination of shadow interaction and SAR, as well as see a use for it in exhibition environments.

Schraffenberger, H. & van der Heide, E. (2013a). From Coexistence to Interaction: Influences Between the Virtual and the Real in Augmented Reality. In K. Cleland, L. Fisher, & R. Harley (Eds.), Proceedings of the 19th International Symposium on Electronic Art, ISEA2013 (pp. 1–3). Sydney.

Abstract: This paper examines the interaction and influences between the virtual and the real in Augmented Reality (AR). We explore how real objects can affect virtual objects and vice versa. Our work is based on heoretical considerations, a review of existing research and artworks as well as our own initial series of experiments. We argue that virtual and real objects can not only simulate influences that exist between real entities, but also influence each other in new and imaginary ways that have no equivalent in the physical world.

Schraffenberger, H. & van der Heide, E. (2013b). Towards Novel Relationships between the Virtual and the Real in Augmented Reality. In G. De Michelis, F. Tisato, A. Bene, & D. Bernini (Eds.), Arts and Technology (Vol. 116, pp. 73–80). Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering. Springer Berlin Heidelberg.

Abstract: In Augmented Reality (AR), virtual and real content coexist in the same physical environment. However, in order to create AR, solely adding virtual content to a real space does not suffice. In this paper we argue that an augmentation adds and relates something virtual to something real. Subsequently, we discuss both existing and promising future relationships between the virtual and the real. We explore what AR is and what it could possibly include from a technology-independent and conceptual point of view. By comparing our take on AR with common manifestations of AR, we identify possible directions for future research and AR (art) works, such as the use of non-visual modalities and the design of novel interactions between the virtual and the real.

Schraffenberger, H. & van der Heide, E. (2012) Interaction Models for Audience-Artwork Interaction: Current State and Future Directions. In Anthony L. Brooks (Ed.), Arts and Technology – Second International Conference, ArtsIT 2011, Esbjerg, Denmark, December 10-11, 2011, Revised Selected Papers, LNICST 101, Springer.

Abstract: Interactive art is of great relevance to the arts, sciences and technology alike. A common field of interest among researchers of different disciplines, practising artists and art institutes is the interaction between audience and artwork. This paper reviews existing research concerning interaction in interactive art and discusses its applicability for describing and classifying audience-artwork interaction. In pointing out possible future directions, we identify a need for models describing the relation between the audience’s and artwork’s actions and reactions as well as the necessity for future research looking at interaction as a continuous bi-directional process between work and audience.

de Jong, S., Jillissen, J., Kirkali, D., de Rooij, A., Schraffenberger, H. & Terpstra, A. (2010). One-press control: A tactile input method for pressure-sensitive computer keyboards, extended abstract at 28th ACM Conference on Human Factors in Computing Systems (CHI 2010), April 2010, Atlanta (USA).

Abstract: This work presents One-press control, a tactile input method for pressure-sensitive keyboards based on the detection and classification of pressing movements on the already held-down key. To seamlessly integrate the added control input with existing practices for ordinary computer keyboards, the redefined notion of virtual modifier keys is introduced. A number of application examples are given, especially to point out a potential for simplifying existing interactions by replacing modifier key combinations with single key presses. Also, a new class of interaction scenarios employing the technique is proposed, based on an interaction model named “What You Touch Is What You Get (WYTIWYG)”. Here, the proposed tactile input method is used to navigate interaction options, get full previews of potential outcomes, and then either commit to one or abort altogether – all in the space of one key depress / release cycle. The results of user testing indicate some remaining implementation issues, as well as that the technique can be learned within about a quarter of an hour of hands-on operating practice time.

Journals

Haselager, P., Schraffenberger, H., Thill, S., Fischer, S., Lanillos, P., van de Groes, S., & van Hooff, M. (2023). Reflection machines: Supporting effective human oversight over medical decision support systems. Cambridge Quarterly of Healthcare Ethics (p. 1–10). Cambridge University Press.

Abstract: Human decisions are increasingly supported by decision support systems (DSS). Humans are required to remain “on the loop,” by monitoring and approving/rejecting machine recommendations. However, use of DSS can lead to overreliance on machines, reducing human oversight. This paper proposes “reflection machines” (RM) to increase meaningful human control. An RM provides a medical expert not with suggestions for a decision, but with questions that stimulate reflection about decisions. It can refer to data points or suggest counterarguments that are less compatible with the planned decision. RMs think against the proposed decision in order to increase human resistance against automation complacency. Building on preliminary research, this paper will (1) make a case for deriving a set of design requirements for RMs from EU regulations, (2) suggest a way how RMs could support decision-making, (3) describe the possibility of how a prototype of an RM could apply to the medical domain of chronic low back pain, and (4) highlight the importance of exploring an RM’s functionality and the experiences of users working with it.

Cornelissen, N.A.J., van Eerdt, R.J.M., Schraffenberger, H.K., & Haselager, W.F.G. (2022). Reflection machines: increasing meaningful human control over decision support systems. Ethics and Information Technology.

Abstract: Rapid developments in Artificial Intelligence are leading to an increasing human reliance on machine decision making. Even in collaborative efforts with Decision Support Systems (DSSs), where a human expert is expected to make the final decisions, it can be hard to keep the expert actively involved throughout the decision process. DSSs suggest their own solutions and thus invite passive decision making. To keep humans actively ‘on’ the decision-making loop and counter overreliance on machines, we propose a ‘reflection machine’ (RM). This system asks users questions about their decision strategy and thereby prompts them to evaluate their own decisions critically. We discuss what forms RMs can take and present a proof-of-concept implementation of a RM that can produce feedback on users’ decisions in the medical and law domains. We show that the prototype requires very little domain knowledge to create reasonably intelligent critiquing questions. With this prototype, we demonstrate the technical feasibility to develop RMs and hope to pave the way for future research into their effectiveness and value.

Graßl, P., Schraffenberger, H., Borgesius, F., & Buijzen, M. (2021). Dark and bright patterns in cookie consent requests. Journal of Digital Social Research.

Abstract: Dark patterns are (evil) design nudges that steer people’s behaviour through persuasive interface design. Increasingly found in cookie consent requests, they possibly undermine principles of EU privacy law. In two preregistered online experiments we investigated the effects of three common design nudges (default, aesthetic manipulation, obstruction) on users’ consent decisions and their perception of control over their personal data in these situations. In the first experiment (N = 228) we explored the effects of design nudges towards the privacy-unfriendly option (dark patterns). The experiment re- vealed that most participants agreed to all consent requests regardless of dark design nudges. Unexpectedly, despite generally low levels of perceived control, obstructing the privacy-friendly option led to more rather than less perceived control. In the second experiment (N = 255) we reversed the direc- tion of the design nudges towards the privacy-friendly option, which we title “bright patterns”. This time the obstruction and default nudges swayed peo- ple effectively towards the privacy-friendly option, while the result regarding perceived control stayed the same compared to Experiment 1. Overall, our findings support the notion that the EU’s consent requirement for tracking cookies does not work as intended. Further, we give insights into why this might be the case and recommendations on how to address the issue.

Hoffman, A. S., Jacobs, B., van Gastel, B., Schraffenberger, H., Sharon, T., & Pas, B. (2020). Towards a seamful ethics of covid-19 contact tracing apps? Ethics and Information Technology.

Abstract: In the early months of 2020, the deadly Covid-19 disease spread rapidly around the world. In response, national and regional governments implemented a range of emergency lockdown measures, curtailing citizens’ movements and greatly limiting economic activity. More recently, as restrictions begin to be loosened or lifted entirely, the use of so-called contact tracing apps has figured prominently in many jurisdictions’ plans to reopen society. Critics have questioned the utility of such technologies on a number of fronts, both practical and ethical. However, little has been said about the ways in which the normative design choices of app developers, and the products that result therefrom, might contribute to ethical reflection and wider political debate. Drawing from scholarship in critical design and human–computer interaction, this paper examines the development of a QR code-based tracking app called Zwaai (‘Wave’ in Dutch), where its designers explicitly positioned the app as an alternative to the predominant Bluetooth and GPS-based approaches. Through analyzing these designers’ choices, this paper argues that QR code infrastructures can work to surface a set of ethical–political seams, two of which are discussed here—responsibilization and networked (im)permanence—that more ‘seamless’ protocols like Bluetooth actively aim to bypass, and which may go otherwise unnoticed by existing ethical frameworks.

Schraffenberger, H. & van der Heide, E. (2015). Audience-Artwork Interaction. International Journal of Arts and Technology.

Abstract: Interactive art is of great relevance to the arts, sciences and technology alike. A common interest among researchers of different disciplines, practising artists and art institutes is the interaction between audience and artwork. This paper introduces the term audience-artwork interaction to facilitate research in this field and raises the question: how can we describe the interaction between audience and artwork? We discuss relevant research such as definitions and taxonomies of interaction, classifications of interactive systems and audience behaviour studies. A working definition of audience-artwork interaction is established. Furthermore, a model which describes interaction as a dialogue between the audience and a dynamic work is presented. We identify possible future directions and point out the need for models which describe the audiences and artworks actions and reactions as well as the relationships among them.

Schraffenberger, H., & van der Heide, E. (2014). Everything Augmented: On the Real in Augmented Reality. Journal Of Science And Technology Of The Arts, 6(1), 17-29.

Abstract: What is augmented in Augmented Reality (AR)? In this paper, we review existing opinions and show how little consensus exists on this matter. Subsequently, we approach the question from a theoretical and technology-independent perspective. We identify spatial and content-based relationships between the virtual and the real as being decisive for AR and come to the conclusion that virtual content augments that to which it relates. Subsequently, we categorize different forms of AR based on what is augmented. We distinguish between augmented environments, augmented objects, augmented humans and augmented content and consider the possibility of augmented perception. The categories are illustrated with AR (art) works and conceptual differences between them are pointed out. Moreover, we discuss what the real contributes to AR and how it can shape (future) AR experiences. A summary of our findings and suggestions for future research and practice, such as research into multimodal and crossmodal AR, conclude the paper.

Editorial Work

Schraffenberger, H., Kniveton, M., Kolstee, L., Kolstee, Y., Verlinden, J., & Wesdorp, R. (Eds.) (2014, May). AR[t], Augmented Reality, Art and Technology, issue 5.
Schraffenberger, H., Kniveton, M., Kolstee, Y., Verlinden, J., & Wesdorp, R. (Eds.) (2013, November). AR[t], Augmented Reality, Art and Technology, issue 4.
Schraffenberger, H., Kniveton, M., Kolstee, Y., & Verlinden, J. (Eds.) (2013, May). AR[t], Augmented Reality, Art and Technology, issue 3.
Kolstee, Y., Schraffenberger, H., Vahrmeijer E., & Verlinden, J. (Eds.) (2012, November). AR[t], Augmented Reality, Art and Technology, issue 2.
Kolstee, Y., Schraffenberger, H., Vahrmeijer E., & Verlinden, J. (Eds). (2012, April). AR[t], Augmented Reality, Art and Technology, issue 1.

Books

Schraffenberger, H. (2011). Whether you like art or not. Short essays on the liking of modern art. Supervised by Edwin van der Heide. Self-published.

Posters

Kripe, Z., Schraffenberger H., & Terpstra, A. (2010). Interactive Art and Persuasive Technology: The Case Study of the Formamat, Poster presented at the 5th International Conference on Persuasive Technology (Persuasive 2010), June 2010, Copenhagen.

Video Publications

Kripe, Z., Schraffenberger H., & Terpstra, A. (2010). The Formamat: Investigating the Dispensability of Pervasive Data, the 8th International Conference on Pervasive Computing (Pervasive 2010), May 2010, Helsinki. Video published on the USB-proceedings.

Formamat from Hanna Schraffenberger on Vimeo

Magazine Articles

Jacobs, B., & Schraffenberger, H. (2020). Friction for privacy. why privacy by design needs user experience design. In European Cyber Security Perspectives 2020, 12–14.
Schraffenberger, H. & Verlinden, J. (2014, May). The great pig in the sky. In AR[t], Augmented Reality, Art and Technology, 5, ISSN: 2213-2481.
Schraffenberger, H. (2012, November). Chasing virtual spooks, losing real weight. In AR[t], Augmented Reality, Art and Technology, 2, ISSN: 2213-2481.
Schraffenberger, H. (2012, April). Interview with Helen Papagiannis. In AR[t], Augmented Reality, Art and Technology, 1, ISSN: 2213-2481.
Schraffenberger, H. (2012, April). Lieven van Velthoven — the racing star. In AR[t], Augmented Reality, Art and Technology, 1, ISSN: 2213-2481.

Newspaper Articles

Schraffenberger, H. (21 April 2011). Kunst? Welke kunst? In Mare – Leids Universitair
Weekblad, no. 27.

Online Columns and Blog Posts

Schraffenberger, H. (2021). Pretty verifier names. IRMA made easy (blog).
Schraffenberger, H. (2021). Looking back, looking forward. IRMA made easy (blog).
Schraffenberger, H. (2021). Facilitating a quick start: Exploring possibilities for an IRMA wizard. IRMA made easy (blog).
Schraffenberger, H. (2021). A big step forward: Expert review of the new IRMA app. IRMA made easy (blog).
Schraffenberger, H. (2021). 12345 is not secure: Proposing a PIN check feature for IRMA. IRMA made easy (blog).
Block, G., and Schraffenberger, H. (2021). IRMA translations made easy. IRMA made easy (blog).
Schraffenberger, H. (2021). IRMA made accessible: first steps. IRMA made easy (blog).
Schraffenberger, H. (2021). IRMA made accessible: can you read this? IRMA made easy (blog).
Schraffenberger, H. (2021). Assessing and improving the accessibility of the IRMA documentation. IRMA made easy (blog).
Schraffenberger, H. (2021). What is IRMA – and why do we need to make it easy? IRMA made easy (blog).
Schraffenberger, H. (2021). The IRMA site in a new guise: improving IRMA’s web presence. IRMA made easy (blog).
Schraffenberger, H. (17th May 2012). What matter does! Blog of the DEAF festival 2012.
Schraffenberger, H. (18th May 2012). The beauty of a flower. Blog of the DEAF festival 2012.