Conclusion
The data generation phase of this project has concluded, despite the fact that my learning experience with this system and the concepts related to it will undoubtedly continue for years to come. Not only will this system continue to grow leading to my Human Theremin Project exhibition with Lauren in December, but this adventure of embodiment, pre-cognitive interaction and interactive improvisation may define my creative career for some time now, considering the reach and impact this project has had. Thank you for reading the blog, if you are; several people have approached me in recent weeks to dumbfound me with the news that they’ve engaged with these musings. Below is the final studio demonstration of the system, complete with finalised visuals, live processed Zither, body visualisation, drum-synced visual sampling and, of course, body controlled modular synthesis.
Neatly wrapping up the data generation phase is a performance I completed at local experimental venue The Cave Inn. This opportunity provided a very convenient and realistic example of the system in a typical gig setting. A quick load in, a tiny stage packed with the other acts’ setups, edged up against a drum kit, no projection surface - a perfect reminder of what it really is to be a gigging musician. With all my time away from bands, I had forgotten the feeling. However, the system adapted to a very high standard. After a quick mapping session back in the studio during which I recorded a handful of extra examples into Wekinator; refining some sonic scenes close to the ground and in a kneeling position, I had a relatively cohesive range of sonic palettes accessible through a movement sequence that began kneeling, slowly moved up and shifted between left and right ‘warrior’ yoga poses. Notably pleasing sonic palettes also arose from a forward bend pose and various wide leg / wide arm poses; all of which were discovered through attentive improvisation. Due to the limited space on stage, I was forced to explore more vertical range with my body than horizontal, but this did not necessarily limit the performance. By slowly exploring the range of options available in kneeling, squatting and standing I found an adequate range of musical scenes, however ambient. The greatest takeaway from this experience in particular was the effectiveness of simply recording new examples into the Wekinator project I have been using thus far to refine system use ahead of a show. At first, I imagined I may have to re-record all examples to prevent a jumpy output, however extrapolation remained smooth with 3-4 new examples recorded across the models.
Visually, the piece was polished up with the addition of PBR materials applied to the particle and body geometries. Referencing a glass material and used with an environment light (referencing a studio HDR image), the visuals achieved a drastically more realistic feel and appear of a much higher professional quality. Balancing the opacity of the PBR with the underlying colour of the particles was the key to having them ‘pop’ against a black backdrop. See below for the recording of the set!
Answering the Question
How can movement interaction extend audiovisual performance with a modular synthesiser?
Okay, here goes. The following is written in an inspired state of stream of consciousness.
Movement interaction may extend audiovisual performance with a modular synthesiser by giving the performer a heightened awareness of the sound they are producing and their relationship to it. Movement interaction may limit the distance between the performer and their music to an incredible degree, in which any movement of the body whatsoever will have at least some sonic effect. Of this nature, there is no instrument yet. By engaging interactive machine learning techniques, this phenomenon may be developed further by creating an interaction in which sonic results are not necessarily predictable - at least until the user develops familiarity with the system. Perhaps it is better to say that the interactions are not obvious - or essentially intuitive. However, the system will react the same way to the same pose / movement each time it is used, so the interactions may be learnt - not intellectually learnt, but known through a process of embodied learning (Tanaka) to form embodied knowledge. This is essentially a pre-cognitive understanding of the sound and the body’s relationship to it. Here, we are beginning to touch on the true significance of the instrument. If, on the one hand, we value listening as the genesis to improvisation and attentive reaction as the fundamental element of improvised performance (in the school of Oliveros, Cage, Young) and on the other aspire to create musical interfaces that inspire through immersivity, the Abstract Human Theremin system deserves deep consideration.
Okay, back to the point - but why does this extend the performance? I think, because it extends the performer. We can look at audience reaction, audience engagement, physicality, convincing interaction, blah blah, but this study did not involve any quantitative audience data, so to steer clear of conjecture I will cast that realm from the scope. If anything, in watching myself perform, I think the level of quality added to the performance in the audience’s view is highly dependent on the performativity of the physicality - which is essentially choreography. So, for audience purposes, the system is best used in collaboration with a dancer - which in itself is revolutionary, let’s be honest.
Back to the point - again - it extends the performer. It extends the performer’s immersion in the music and challenges the performer to be completely enamoured within the sound. There is no second guessing, no escape, no hiding - only listening to the sound you are interacting with, deciding whether a change is required, then either moving to a predictable pose or attentively exploring the unknown. Like walking through a rainforest in the dark, the experience is captivating. Add into this the reactive visual element, and you are provided abstract heuristics for movement exploration - you move, the particles sway, you sway, the particles spin, you spin etc. etc. It is an utterly magical way to perform. Does anyone want to watch? That’s up to them. In my experience, funnily enough, yes.
And how the system changes composition! Turns it on its head and scrambles the contents on the floor. In the tradition of Mumma (1967), system design becomes the primary mode for composition. Fitting entire pieces into infinite combinations of one to five volts over sixteen values. Where can the piece go? The scary part is, move too quickly, and show all your cards. Again, pushing the performer - move slower, listen deeper, what is actually happening in the sound?
This is the experience I have had with the system. Open minded adventure and fresh ideas drawn from discourse have led me to this articulation, a neat new chapter in considering human computer interaction, embodied improvisation and abstract multi-media performance. Again, our world is becoming so digitised, but we don’t need to leave the body behind. Our bodily awareness is innate and incredible, outputting such a wide range of ‘data’ (if we need to put it that way) for a computer to interpret with such comparative ease on our end to produce it. When we use the body, we may be embodied, unlocking new levels of intuition and improvisation. Create symphonies of sound simply by moving. This is the greatest interface we have, and it is up to artist designers like me/us to include it in the mediums of the future.
Wrapping with an App
This week, I applied to install and/or perform this system as a part of ISEA 2024, an international conference on electronic art coming to Brisbane next year. I think that my thematic explanations of the system written for the application make for a fitting evaluation of the system on which to close. I titled the installation ‘Play by Hear’ for this application.
“This piece, ‘Play by Hear’, aims to cultivate awarenesses in its users through audiovisual movement interaction. The experience is designed to sensitise one’s ears to sonic subtlety and refocus one’s bodily awareness to the impacts we may have on our environment, achieved through refined control of sound and visual materials. Users move slowly to interact with the system, attentive to the subtle manipulations in sonic and visual material that their movements create. Their movements may not always have the anticipated impact, however the system will repeat itself to the same movement, allowing users to form an embodied understanding of how it may be performed intuitively. Beginning with field recordings taken from around Australia and Indonesia, the user modulates resonators, reverbs and other effects to transform natural environments into captivating music soundscapes; from slow, haunting arpeggios emanating from croaks among nighttime rice-paddies to divine reprisals of euphoric chords struck by morning bird calls. With musical form delegated to the natural world, this piece pays homage to environmental sound as the genesis of all sonic practice, participating in the Indian conception of music as a perpetual song to which the musician’s role is to decorate rather than distract (as described by Sri Ramakrishna). These changes in sonic composition through movement bring a heightened awareness to both the ear and the body, blurring sonic lines between the natural experience and art. Similarly, the work blurs lines between the ‘real’ and digital, seamlessly welcoming the body into a hyper-real environment in which the total corpus becomes the interface for electronic interaction. In a world becoming increasingly digital, this work reimagines the role that our physical existence may play by extending our physically creative ability rather than limiting it. To create with an open mind, to create intuitively, to create in responsive commentary to the natural world around you, is to participate in the tradition of the cosmos. To join this undying and unbroken line from the primordial storm that unceasingly reconfigures itself into all that exists is to become nature; to experience the nature of nature. Engaging with this thematic undercurrent of ‘Everywhen’, ‘Play by Hear’ provides an opportunity for everyday attendees to gain a pre-cognitive, embodied insight into the dialectic nature of creation. Through interaction, the essences of mutual causality and inter-dependent creation may be intimately understood in this piece.”
First Draft of Findings (Word Jam)
Movement interaction may extend audiovisual performance with a modular synthesiser by giving the performer a heightened awareness of the sound they are producing and their relationship to it. Movement interaction may limit the distance between the performer and their music to an incredible degree, in which any movement of the body whatsoever will have at least some sonic effect. Of this nature, there is no instrument yet. By engaging interactive machine learning techniques, this phenomenon may be developed further by creating an interaction in which sonic results are not necessarily predictable - at least until the user develops familiarity with the system. Perhaps it is better to say that the interactions are not obvious - or essentially intuitive. However, the system will react the same way to the same pose / movement each time it is used, so the interactions may be learnt - not intellectually learnt, but known through a process of embodied learning (Tanaka) to form embodied knowledge. This is essentially a pre-cognitive understanding of the sound and the body’s relationship to it. Here, we are beginning to touch on the true significance of the instrument. If, on the one hand, we value listening as the genesis to improvisation and attentive reaction as the fundamental element of improvised performance (in the school of Oliveros, Cage, Young) and on the other aspire to create musical interfaces that inspire through immersivity, the Human Theremin system deserves deep consideration.
Okay, back to the point - but why does this extend the performance? I think, because it extends the performer. We can look at audience reaction, audience engagement, physicality, convincing interaction, blah blah, but this study did not involve any quantitative audience data, so to steer clear of conjecture I will cast that realm from the scope. If anything, in watching myself perform, I think the level of quality added to the performance in the audience’s view is highly dependent on the performativity of the physicality - which is essentially choreography. So, for audience purposes, the system is best used in collaboration with a dancer - which in itself is revolutionary, let’s be honest.
Back to the point - again - it extends the performer. It extends the performer’s immersion in the music and challenges the performer to be completely enamoured within the sound. There is no second guessing, no escape, no hiding - only listening to the sound you are interacting with, deciding whether a change is required, then either moving to a predictable pose or attentively exploring the unknown. Like walking through a rainforest in the dark, the experience is captivating. Add into this the reactive visual element, and you are provided abstract heuristics for movement exploration - you move, the particles sway, you sway, the particles spin, you spin etc. etc. It is an utterly magical way to perform. Does anyone want to watch? That’s up to them. In my experience, funnily enough, yes.
And how the system changes composition! Turns it on its head and scrambles the contents on the floor. In the tradition of Mumma (1967), system design becomes the primary mode for composition. Fitting entire pieces into infinite combinations of one to five volts over sixteen values. Where can the piece go? The scary part is, move too quickly, and show all your cards. Again, pushing the performer - move slower, listen deeper, what is actually happening in the sound?
This is the experience I have had with the system. Open minded adventure and fresh ideas drawn from discourse have led me to this articulation, a neat new chapter in considering human computer interaction, embodied improvisation and abstract multi-media performance. Again, our world is becoming so digitised, but we don’t need to leave the body behind. Our bodily awareness is innate and incredible, outputting such a wide range of ‘data’ (if we need to put it that way) for a computer to interpret with such comparative ease on our end to produce it. When we use the body, we may be embodied, unlocking new levels of intuition and improvisation. Create symphonies of sound simply by moving. This is the greatest interface we have, and it is up to artist designers like me/us to include it in the mediums of the future.
”
