VR Sensory Soundscapes
Virtual Reality Experience
Jan - May 2023
VR sensory soundscapes was created as part of my Honours project to research music visualisation techniques within virtual reality experiences. The project offers a unique, interactive and fully multisensory experience that transports participants through three musical environments. Each environment explores a different musical genre and music visualisation technique. The project was created in Unreal Engine 5 and uses the Oculus Quest 2 headset.
Prior to creation, the design process involved several stages, including research, ideation and user testing. Extensive research was conducted exploring the psychological impact of sound/visual stimuli on human perception and case studies were carried out exploring the representation of musical genres. This research informed the ideation phase which was used to guide the creation of the virtual environments and soundscapes. User testing was used throughout all stages of development to gain feedback from users on design and functionality.
Unreal Engine 5 was utilised for the project’s creation making use of its real-time lighting features to produce the virtual environments. RipX’s AI features were used to isolate individual stems from musical compositions to extract each instrument allowing these sounds to be placed individually within the virtual environments to allow stereoscopic sound implementation. The project also leveraged the capabilities of the Oculus Quest 2 headsets and controllers to ensure a seamless and immersive experience for users.
The experience was fully multi-sensory with the headset accounting for sight and sound, the controllers haptic feedback eliciting touch, real life use of diffusers and aromas were used to involve smell and the addition of mocktails for taste.
Poster Presentation Day
Music Visualisation Techniques
1
Environment Design
Within the reggae environment, music was visualised through environment design. This was achieved through analysing the visual culture of the music genre through a case study and implementing this into the virtual environment's design with use of meshes, particle systems and textures. The sense of presence was found to be the strongest within this environment with an increase of emotional reactions. With greater importance being placed on realism through this technique, users relaxed, believing the environment around them instead of probing it which resulted in users better embodying the experience.
2
Interactivity
The jazz bar explored musical interactivity, allowing user input to change the sounds heard in the environment in real-time. For example, users could pick up a saxophone, interacting with input buttons and then hearing greater emphasis on the saxophone's part in the music. This technique resulted in strong engagment from users with participants reporting feeling a flow like state and having more fun through interacting with their surroundings and hearing the impacts of this. Interactivity was implemented through the addition of VR hands which were then set up to play responsive animations when collisions occurred with specific meshes.
3
Sonic Animations
The disco environment explored sonic animation through analysing the audio’s soundwaves. This analysis was stored as data which was then later called upon by functions before game play to avoid having a negative impact on performance. The analyser can then be used to utilise several patterns within the soundwaves such as loudness and the strength of individual bands within the audio. These extractions were then used to provide correlating visuals to the user in several ways. For example, animating geometries to change scale in accordance to the changing loudness of a composition. This brings a great sense of musicality to the environments through satisfying users audio-visual expectations.