Consulting Gillies (2019)
Leading on from last week’s brief reflections on Marco Gillies’ 2019 article ‘Understanding the Role of Machine Learning in Movement Interaction’, I have today revisited the text in full. Unpacking the design tenets of movement interaction, its history and the possibilities afforded by new kinds of machine learning, this text is one of the most relevant academic sources I have come across. Gillies’ work has also informed the work of Tim Murray-Browne to a large extent. Since first engaging with the text, my experimentation with movement interaction design techniques have provided me with a wealth of context for Gillies’ ideas through which I have discovered a much deeper understanding of the concepts explored. As the article focuses on interactive machine learning (IML) and includes a case study on Wekinator, my engagement of IML and the software has also afforded an intimate view of the discussion. The article bolsters and elaborates on conclusions I have arrived at myself while outlining exciting concepts that I would like to explore practically, in turn alluding to improvements that could be applied to my thinking. I feel it is important to finish the project with the same tethering to discourse I began it with, so I am excited by how fulfilling this re-engagement has been.
Best Practices / Movement Theory
Gillies begins the article with a close up view of interactive system design by outlining some best practices and justifications that align with my own findings. He stresses the importance of a first-person view of the system throughout its construction, describing how an embodied experience can not easily be analogized or sketched. Designing through system use has been the only way I have been able to truly conceive of and make decisions, which supports this point and highlights the relevance of artistic research to this kind of project.
Within the overview of movement interaction, Gillies also provides reasoning for engaging this form of HCI at all, including an elegant articulation of a fact I realise I had taken for granted. Namely, that using one’s body to interact is simply more emotionally engaging. By citing a wealth of studies and theories such as Damasio’s Somatic Marker Hypothesis (Damasio 1994), Gillies suggests that “movement interfaces can have powerful emotional effects that more traditional interfaces cannot” (p. 9). However, Gillies qualifies this by stating “if it makes use of our embodied knowledge and reflection” (p. 9). While this notion is undoubtedly supported by my experiences, the concepts of ‘embodied knowledge’ and ‘reflection’ need clarification. Thankfully, Gillies explores this in depth by describing the roles that cognitive thought and tacit understandings play in movement and, thus, movement interaction.
Gillies’s explanations of embodied cognition, embodied knowledge and motor learning form a detailed picture of how we accumulate, use and reflect on movement that is neither totally tacit nor cognitive. After making the case for how embodied or learned actions become once they are ‘automatic’ (like riding a bike), Gillies emphasises just how cognitive movement knowledge can be in the learning process as one attempts to make sense between articulated directions and how to move. While the significance of embodied knowledge in movement interaction has been obvious to me from the start, the likelihood for a cognitive, intellectual experience to take hold when one first uses the system is a phenomenon I have not accounted for. In fact, in reflection, it explains some of the discomfort I believe to have witnessed when friends have tried using the system. They’ve often opened with awkward, uncomfortable movements, unsure of what is expected or best. As I understand and use the system often, my experience easily becomes one of bodily awareness and deep sensory attention - how can the user be encouraged to arrive at this point early on? How can the user be encouraged not to try to ‘figure out’ the system, but instead treated playfully and exploratively?
Somatic Reflection
An answer to this may come from Gillies’ exploration of Somatic Reflection; a concept that has emerged from the deeper understandings of HCI provided by authors such as Schiphorst (2009, 2011) Nuñez Pacheco & Loke (2018) and Höök and associates (2015, 2016). In this understanding, Somatic Reflection is based on Shusterman (2008)’s philosophy of Somaesthetics, which explores cognition through the body using yoga, meditation and Feldenkrais technique. By engaging these practices of slow movements and drawing attention to the feelings of the body, including discomfort, tension and relief, one may become extraordinarily aware of their body and reflect on their embodied experience.
This form of meditative reflection-in-action was particularly rousing to come across, as it mirrors the way I have relied on yogic or Tai-Chi movements when designing the system on my own and with Lauren. These experiences show significant efficacy for the concept, which I would love to explore further while engaging with the writers described above. Additionally, when combined with Gillies’ suggestion of movement interaction providing the basis for deeper emotional experiences, this potential for increased bodily awareness and reflection forms a very solid support for movement interaction in the arts, summarised in Gillies stating that “movement interaction might, therefore, be able not only to use our embodied skills and knowledge but improve them.” (p. 11) Dovetailing with my passion for meditative music practice in the tradition of La Monte Young and Pauline Oliveros, the Somatic Reflection framework may be a key to unlock new ideas within my design, i.e. techniques or features that encourage inquisitive, attentive and instinctive system use.
Lauren Sherlock completing a sequence of movements informed by Yoga and Tai-Chi to use the system.
Reflection in Action in Interactive Machine Learning
Finally, Gillies arrives at his conceptual model for movement interaction design using IML: Schön’s reflective cycle. Having explored this model deeply in relation to artistic research methodology, I am in a much better position to understand Gillies’ arguments now than when I began this project. Gillies builds on his model of embodied knowledge by highlighting how it can not be taught through articulation nor demonstration alone, but instead requires a complex interaction involving both. By citing Höök's experience of learning English horse-riding, Gillies highlights similarities between motor learning and Schön’s ‘reflective practicum’, revealing that the key to communicating tacit knowledge is a cycle of reflection. In the teacher-student context, this cycle involves the two actors reflecting on each other's actions and their own in order to understand what those actions mean to themselves and the other person - understanding what is meant by “Put your weight in your heels” or understanding how that direction may be misinterpreted, to use the horse-riding example. To bring the subject back to movement interaction design, Gillies substitutes the student for the computer, emphasising how when designing movement interaction we are dealing with knowledge that really must be demonstrated in one way or another. Having detailed this need, the application of interactive machine learning is then obvious: as opposed to normal machine learning, which involves a large corpus of recordings being given to an algorithm to sort through and interpret, interactive machine learning describes a process in which examples are given continuously to refine the reaction of the software. In essence, this is a reflective cycle - one in which the machine learning algorithm is ‘taught’; it uses this knowledge to the best of its ability, then it is provided with another example to become more accurate. Gillies describes this as the reflective cycle between the “designer and artefact’”p. 15).
Marco Gillies’ reflective loop in reference to an Interactive Machine Learning process.
With the understanding of embodied knowledge that Gillies provides, the strengths of interactive machine learning for movement interaction design are spotlighted. Though I was undoubtedly aware of the efficacy the software provides, this articulation of its strength of process makes me want to modify my use of it to focus more on iterative example recording. Until now, my focus has been much more on the output of Wekinator than the input; a process in which I have treated it somewhat more like machine learning rather than interactive machine learning. Now that the output signal flow is operational and close to finalised, I would very much like to channel Gillies’ reminder of the reflective cycle in the final design stages to tune the machine’s interpretation of the body, using IML to the best of its ability.
At the outset of this project, I had intended to be abreast of the discourse throughout the design phase. To some extent, this has been the case - I have explored big-picture ideas through my practice, used them as a guide in system design and consulted various areas of research to solve problems along the way. However, interfacing directly with published work has definitely been eclipsed by practical experiments in the past few weeks. In the final stage of the project now, I am pleased to be revisiting the authors that informed this project’s inception and thrilled to be uncovering deeper understandings of their concepts now that I’ve had some time to contextualise them. This particular re-reading of Gillies has provided insight into what else I might explore and how; ideas that may be the icing on this movement interactive cake. Emotional engagement, Somatic Reflection and Schön are all elements that I am happy to have in the front of my mind during these final touches.
