A Study on the implementation of immersive sound using
multiple speaker systems according to the location of
sound sources in live performance
Kim, Hyo Jun Lee, Seungyon-Seny
SangMyung University SangMyung University
gsm 148@gm a il.com sen ylee01@gm a il.com
Abstract
This paper aims to study the location of sound sources when playing immersive sound using multiple speaker systems in a live performance environment. In particular, this performance designated the visually impaired as the audience. Therefore, the layout design of the sound source was produced by applying immersive technology so that the audience can fully enjoy and immerse in the performance.
The construction of an immersive sound system by installing a number of additional speakers in the existing grand theater performance hall and the effective delivery of immersive sound to the audience are intended in this study. In order to apply the immersive sound technology, an acoustic environment that can provide immersion to the audience by studying the process of selecting and installing the most efficient system and producing sound based on the theoretical background was established.
Through the experiment, the results of the directionality of implementing the location of the immersive sound source using multiple speaker systems are summarized by the location of the sound sources. As a result, the sound source can be positioned anywhere in space, such as outside the installation location of the speaker or inside the listening seat, but it is difficult to recognize the exact location according to the direction. However, due to the characteristics of the plane wave that allows multiple listeners to perceive the same direction of the sound source, the direction of the sound source was the same in all seats of the listening seat.
Keywords: Immersive Sound, Immersive Sound, multiple speaker system, live performance
1. Theoretical Background
Through the development of high-speed communication technology and virtual reality technology, VR
technology is gradually penetrating our lives through services such as virtual experience, VR content, VR game, and VR/AR education [1]. Furthermore, there is a growing interest in high-quality 3D multimedia contents and the supply of contents. With the development of the 3D industry, interest in realistic sound technology that reproduces the three-dimensional effect through sound sources is increasing [2]. In addition, with the
development of game programs, sound technology using headphones is continuously developing, and the embodiment of stereophonic sound using existing stereo headphones is gradually developing as well. Wireless earphones that render spatial sound with dynamic head tracking technology have been released. Movie theaters are also making special sound theaters that go beyond surround sound systems. Even in individual listening environments, the desire for immersive sound system is increasing. Immersive sound is a technology that reproduces the natural sound that we experience every day. It is a technology that enables listeners in a space other than the space where the sound source is generated to perceive the same sense of direction, distance, and space as the space where the sound source is generated [3]. In addition, it refers to a technology
that increases the sense of presence and immersion by grafting 3D images to predict the distance to the sound source in real time. Based on a technology that reproduces the sound of a virtual space, it also refers to a technology that enables responsive interaction according to the movement of the user when turning his or her head or moving a position. However, most concert halls in Korea are currently using stereo sound technology. There is a desperate need to apply a next-generation sound system that can satisfy the needs of the audience even in the concert halls. The term immersive sound is used interchangeably with
stereophonic sound, 3D sound,spatial sound and spatial audio. The terminology of sound technology includes the technology of recording, production, transmission, and reproduction of sound, and it is important to understand and know how to apply this technology accurately. This study intends to collectively refer to the term immersive sound as the most used immersive sound in a performance environment based on live performances.
2 Research Purpose
Among the various acoustic technology methods for reproducing immersive sound is the method using a headphone and speaker system. This study aims to focus on the immersive sound reproduction using a speaker
system, excluding the implementation method using headphones and individual listening environments. The immersive sound using multiple speakers is analyzed by measuring the position of the sound source in 3D impulse during the system installation process. The ambient sound is produced by considering a plan for the arrangement of sound sources through the measured analysis results. Through this process, the mobility and direction according to the optimal positioning of the sound so that the sound source is transmitted to the listener in the intended direction and movement are studied. The performance work for the experiment in this study, ⟪Alfonso Dode's Stars⟫, was produced as a performance for the hearing impaired, and was conducted live at three different local performance venues. Although there may be some differences in the production experiment according to the characteristics of the performance hall, this study aims to focus on the mobility according to the arrangement of the ambient sound of the performance works based on the live performances performed at the Grand Theater of the Iksan Arts Center on October 31, 2020.
Therefore, the objectives of this study are as follows. First, suggest the configuration and installation method of an immersive sound system using multiple speakers in a live performance situation. Second, discuss how sound can be effectively transmitted according to the location of the sound source. Third, present a model for how to deliver a more three-dimensional and immersive sound to the audience than the existing stereo sound system.
3. Performance Production Using
WFS-based Immersive Sound Technology
Sound drama ⟪Alfonso Dode's Star⟫ is an experimental study of ambient sound using immersive sound
technology. Performances in which the disabled are performers are steadily progressing, but there are very few cases of performances in consideration of visually impaired as an audience [4]. Therefore, this work can be considered as a sound drama content production for the immersive sound experience that the visually impaired can fully enjoy, unlike the existing performances where the stage set and performance music were the center.
3.1. Performance Overview
The overall flow of the performance was decided by considering the production of ambient sound when adapting the script from the beginning. Examples of sounds that can show the scene even when there are no actors' lines are as follows. In Scene 1, the sound of the sheep farm environment, the indoor ambient sound in the hut in Scene 2, the sound of rain on the trail in Scene 3, and the sound of a campfire on a windy hill in Scene 4 are ambient sounds. In addition, a virtual sound effect is created to render the image of the star as stated in the title ⟪Alfonso Dode's Star⟫.
Musical instruments such as piano, synthesizer, clarinet, cello, and violin have been played directly in the performance hall. For the effect of the play, a foley
artist directly arranges the position of the foley sound, such as the sound of closing the door, the sound of thunder, and the sound of footsteps. Foley Sound is placed to create the optimal sense of space by experimenting several times on which speaker to be played. <Figure 1> shows the location of musical instruments and sound effects.
Figure 1. Stage Layout of Performers and Actors
Figure 2. Installation of the Foley Artist Tool
3.2. Performance Planning Using Immersive
Sound Technology
When visually impaired people watch a performance, it is difficult to immerse themselves into the performance as much as non-disabled people. Unlike the
conventional performances, the background set and video on the stage are drastically abandoned, and the effect sound that was previously worked with stereo sound is produced as 3D stereophonic sound to design the background sound more like a real background space. In order to faithfully deliver the produced 3D stereophonic sound to the audience in large theater performances and to provide a wide sweet spot, the optimal listening position, a technology that uses multiple speakers is suitable. Therefore, WFS-based system is selected among immersive sound technologies.
3.2.1. immersive sound equipment selection
According to the production plan the performance, a sound effect with 360-degree movement was needed during the live performance. Therefore, among DSP systems that can use WFS-based immersive sound technology in live performances, a system in which real-time sound source movement is possible was needed. In addition, it was essential to select a show control mode and a play mode equipment that can be stored in advance as an immersive sound and played back according to the scene during the performance.
Recently, many professional audio companies around the world are launching immersive sound DSP systems.
Table 1. List of Speakers of the Immersive System
Various products such as L-Acoustics, d&b audio,
Meyer sound, and YAMAHA have been developed.
Among them, L-ISA of L-Acoustics and Sara of Astro
Spatial Audio, which have the essential functions for
this performance, were selected after comparison. Since
Acoustics' ISA must be designed using the same L-Acoustics speaker, Sara of Astro Spatial Audio, which
suits the production environment and installation conditions best, was selected. Sara can be controlled by any device that can be configured with the Dante network, regardless of equipment such as speakers and amplifiers that are configured together. Sara can be controlled by Dante's network-configurable products regardless of equipment such as speakers and amplifiers that are configured together. In addition, Dante's output can output up to 128 channels, which is useful for multi-channel performances such as the one focused in this performance.
3.2.2. System layout Design
In order to deliver the immersive sound, it is necessary to arrange speakers uniformly on five sides, including the front, rear, left and right sides, and ceiling, centering on the auditorium. For this performance, a total of 40 speakers - 7 at the top, 7 at the bottom, 6 at the left and right, 6 at the rear and ceiling, and 2 Sub Woofers- were installed and evenly spaced. The size of the theater is 15m from side to side of the stage and 15m from front to back of the stage, and the audience seats are configured with a distance of up to 20m from side to side and 25m from front to back.
Figure 3. Configuration of the Immersive Sound System
Audio input through the console is tuned using an equalizer and a compressor, and then immersive sound is played through 40 speakers through DPS. The transmission of audio signals between devices is composed of Dante, and the console and DSP are connected through a network hub. The input source enters the Rio3224 via a microphone and an analog cable through a DI-Box. The Dante network hub connects to the main console, monitor console, and Astro SARA. Basic tones were created on the main console and 3D real-time sound sources were controlled in SARA. In the monitor console, the input sound is processed in the console in the same way as in the previous performance, so that actors and performers can monitor with the speakers on the stage.
3.2.3. Speaker Layout Design
The speaker layout design was modeled in 3D and the sound pressure distribution of the entire auditorium was predicted through the simulation process. The result of sound pressure distribution was predicted using AFMG's
EASE simulation program. The results of the sound
pressure distribution are as follows.
Figure 4. Speaker Layout 3D Modeling
Figure 5. Sound Pressure Distribution in the Auditorium
It was confirmed that the sound pressure deviation in the entire auditorium was 2dB based on the broadband, and after installing the speaker, it was possible to listen at a uniform sound pressure at any seat.
3.3. System Installation
The side and rear speakers in the auditorium were installed using a speaker stand at a height according to
Classification manufacturer product
name Quantity unit
1 Main Speaker RCF C5212-96 7 EA 2 Front Speaker RCF C5212-96 7 EA 3 Left Speaker RCF C3108-96 6 EA 4 Right Speaker RCF C3108-96 6 EA 5 Resr Speaker RCF C3108-96 6 EA 6 Ceiling Speaker RCF C3108-96 6 EA
7 Subwoofer Adamson Spektrix
Sub 2 EA
the sitting height of the listener. The ceiling speaker was constructed with a structure that could be fixed and installed by safely lowering it with the wire.
3.4. 3D Impulse Measurement
To design a background sound, several factors must be considered, among which the location of the sound source must be expressed as intended. Through 3D impulse measurement, the directionality of each frequency according to the location of the sound source is objectively analyzed. It is measured through the
AARAE program, a stereophonic sound measurement
program developed for research at the University of Sydney, and the SENNHEISER's AMBEO microphone. How the directionality of each frequency changes when the location of the sound source is located in the front, side, and ceiling is analyzed. The gray circle in the center of the coordinates in the following table represent the listening seat. The dot on the gray circle indicates the speaker's position. The 3D impulse measurement is displayed in three colors. After the sound source is generated, it shows the directionality according to the time difference. The red line is 10ms, the blue line is 50ms, and the green line is 100ms. In order to find out the directionality of each frequency for the low, mid, and high frequencies, measurements were made based on 250Hz, 1000Hz, and 4000Hz. The directions of blue and yellow green were regarded as including spatial reverberation, and the analysis was centered on the direction of red.
Figure 6. Installation of Side Speaker(above), Upper front Speaker and Ceiling Speaker(left)
Installation of Speakers in the Auditorium(right)
3.4.1. Measurement Analysis of the Front
When the virtual sound source is located in front, how the directionality of each frequency changes as the distance from the listener's position increases is analyzed. The measurement positions at the front were measured at three points separated by 15m from the front center of the listening seat. The following are the three measurement locations and coordinates on the front.
① Directionality Analysis by Frequency at P1 Position The measurement position of the P1 sound source (X 0m, Y 12.5m, Z 0m) is the front center of the listening seat.
As a result of analyzing the red line measured within 10ms from the position of P1, it has a more pronounced directionality in the low range of 250Hz. In the middle and high frequencies of 1000Hz and 4000Hz, there is a directivity in the front, but the low range has a more pronounced directionality.
Table 2. 3D measurement result at the position of P1
Table 3. 3D measurement result at the position of P2
② Directionality Analysis by Frequency at P2 Position The measurement position of the P2 sound source (X 0m, Y 27.5m, Z 0m) is 15m away from the front center
of the listening seat. As shown above, even at the position of P2 (X 0m, Y 27.5m, Z 0m), the same as in
the position of P1 (X 0m, Y 12.5m, Z 0m), the low P2. X 0m, Y 27.5m, Z 0m 250 Hz
1000 Hz 4000 Hz
P1. X 0m, Y 12.5m, Z 0m 250 Hz
range has more pronounced directionality than the high range. Comparing the measurement results of P1 and P2, it can be seen that the farther the sound source is, the more the sound source exists like a plane from the front rather than the exact location.
③ Directionality Analysis by Frequency at P3 Position The measurement position of the P3 sound source (X 0m, Y42.5m, Z 0m) is located 30m away from the front center of the listening seat. P3, like the positions of P1 and P2, has a more pronounced directionality in the low range than in the high range. Comparing the
measurement results of P2 and P3, if the sound source is twice as far as the distance between the center of the listening seat and the center of the front, the frequency-specific characteristics of the sound source do not change significantly. Therefore, when the location of a sound source must be accurately expressed, the effect cannot be achieved as the distance increases.
Table 4. 3D measurement result at the position of P3
3.4.2. Side measurement analysis
How the directionality of the virtual sound source varies by frequency in front and behind the side is analyzed. Based on the fact that the left and right sides were the same, only the right side was measured, and the measurement position of the side was measured at three points: 5m in front of the center, 5m in the middle, and back.
① Directionality Analysis by Frequency at P4 position The measurement position of the P4 sound source (X 20m, Y 5m, Z 0m) is 15m to the right and 5m away from that location.
Contrary to the location of the sound sources of P1, P2, and P3, it can be seen that the high range has a more pronounced directionality than the low range.
Comparing the measurement results of P1 and P4, it can be seen that the position of the low range is more effective in the front of the sound source and the
position of the high range is more effective from the side.
Table 5. 3D measurement result at the position of P4
② Directionality Analysis by Frequency at P5 position The measurement position of the P5 sound source (X 20m, Y 0m, Z 0m) is 15m away from the center to the right from the center of the audience seat.
It can be seen that even at the position of P5, the high range has a more pronounced directionality. Comparing the measurement results of P4 and P5, it can be seen from the side that a more definite position is expressed when the position of the sound source moves from front to center.
Table 6. 3D measurement result at the position of P5
③ Directionality Analysis by Frequency at P6 position The measurement position of the P6 sound source (X 20m, Y -5m, Z 0m) is a position 15m away from the center to the right and 5m away from the center of the audience seat.
It can be seen that even at the position of P6, the high P3. X 0m, Y 42.5m, Z 0m 250 Hz 1000 Hz 4000 Hz P4. X 20m, Y 5m, Z 0m 250 Hz 1000 Hz 4000 Hz P5. X 20m, Y 0m, Z 0m 250 Hz 1000 Hz 4000 Hz
range has a more pronounced directionality. However, when comparing the measurement results of P5 and P6, it can be seen that when the position of the sound source on the side moves from the center to the back, the expression of the directionality decreases.
Table 7. 3D measurement result at the position of P6
3.4.3. Height Measurement Analysis
How the directionality of the virtual sound source varies by frequency in the central ceiling and floor is analyzed. When the location of the sound source enters the audience and is placed on the floor, how the directionality of each frequency is expressed was measured. The height measurement position was measured at a height of 9m on the floor and ceiling, centered on the center of the listening seat.
Table 8. 3D measurement result at the position of P7
① Directionality Analysis by Frequency at P7 position The measurement position of the P7 sound source (X 20m, Y 0m, Z 0m) is the location of the center floor after entering the audience seat. When the location of
the sound source enters the audience and is placed on the floor, the low range has the directionality of the floor, but in the high range it’s expressed by the directionality of front rather than the floor.
② Directionality Analysis by Frequency at P8 position The measurement position of the P8 sound source (X 20m, Y 0m, Z 9m) is the ceiling position raised 9m from the center of the audience seat. When the position of the sound source is above the head, it is difficult to have a valid meaning for directionality in all sound ranges.
Table 9. 3D measurement result at the position of P8
3.4.4. Measurement Analysis Result
Analyzing the directionality of each frequency with 3D impulse measurement at an 8-point position centered on the audience seat, the low range has more positive directionality than the high range in the front. In addition, the farther the sound source is from the front, the more the sound source exists as a plane rather than the exact location. Therefore, in order to accurately express the location of the sound source from the front, it should be located close to the location of the listener. The sides, contrary to the front, has a more pronounced directionality in the high range than the low range. In addition, when the position of the sound source from the side moves from the front to the center, a more definite position is expressed. However, when the position of the sound source moves backward from the side, the expression of the direction is declined. When the location of the sound source enters the location of the audience and is located on the center floor, the low range has the directionality of the floor, but in the high range it is expressed in the directionality of the front rather than the floor. Furthermore, when the location of the sound source is above the head, it is difficult to have an effective meaning for directionality in all bandwidth. When the sound source is located above the head, the sound source is expressed entirely on the ceiling surface rather than the exact directionality.
P7. X 20m, Y 0m, Z 0m 250 Hz 1000 Hz 4000 Hz P6. X 20m, Y -5m, Z 0m 250 Hz 1000 Hz 4000 Hz P8. X 20m, Y 0m, Z 9m 250 Hz 1000 Hz 4000 Hz
3.5. Sound Design for Performances
A background sound was produced based on the 3D impulse measurement analysis result. In the
performance of ⟪Alfonso Dode's Star⟫, the background is a flock of 1 sheep, 2 indoor huts, 3 rainy trails, and 4 windy hills with bonfires. The goal of background sound design is to make the audience feel as if they are immersed in the space.
3.5.1. Background Sound Design
Background sound design is more about the presence of a natural sound source rather than an accurate
directional expression. Depending on the 3D impulse measurement result, when arranging sound sources with accurate high-pitched sounds such as a positive cry, the position was moved away from the front side, and the side was placed at the rear of the center, and on the ceiling and the back side to reduce the directional expression and make the sound source entirely audible. When the same sound source is placed in multiple directions, the sound source sounds different depending on where it is placed. In addition, it was designed to produce a real-like background sound by overlapping the same sound unevenly in all directions and placing similar but different crying sounds. Sound sources of nature with indefinite expression of sound image, such as the sound of wind, flowing water, and indoor sound, are designed to be evenly heard in listener's places by diffusing the speakers in all directions.
3.5.2. 360-degree Moving Sound Design
With a 360-degree moving sound effect that could not be expressed in a stereo sound system, the audience can clearly experience the changed system.
As stated in the title of ⟪Alfonso Dode's Star⟫, sound effects related to the sound of stars are focused and produced. In order to effectively express the sound of falling stars, the sound of falling stars and the sound of the scattering like fireworks at the moment it last falls are separately produced. The motion of the stars falling during the performance is controlled in real time at 360 degrees, and the final scattering sound is placed at the top of the center of the front window to create a sound as if it is pouring from the sky. The most important factor when playing a 360-degree moving sound effect in real time is the trajectory of the sound source. The basic 360-degree trajectory is as follows. A more natural movement of the sound source is completed by moving it according to the installation position of the speaker rather than completing a 360 degree by drawing a trajectory according to the position of the listening seat. The sound effect of the stars is located at the height of the theater's physical ceiling, which is 9m from the center of the listening seat, and a sound effect that cannot be felt in the existing sound technology is designed.
3.5.3. Example of Immersive Sound Mixing
From ⟪Alfonso Dode's Star⟫ performance, the sound effects, real-time performances, and actors' voices,
multi-channels are all placed and designed in an immersive environment. The input source channels and arrangements for this performance are as follows. Among the input sound sources, the actor's voice was placed in the front center as much as possible so that the dialogue can be clearly transmitted. Among the channels that can be played in real time, the piano and
synthesizer were arranged on both sides so that stereo input can be optimally expressed. The clarinet and violin are placed in the center above the ceiling so that they can be heard as a whole. Since the cello is an instrument responsible for bass, it was designed to feel resonating from the bottom by placing it on the center floor. Since the sound sources that the foley artist generates in real time on the stage can be heard directly in front of the stage, the sound transmitted from the microphone and played through the speaker can feel awkward if the sound is placed in a completely different position from the direct sound. Therefore, it was placed in the same direction as the sound. Whenever the scene changes, the pre-produced and stored background sound with 32 channels of immersive sound was played according to the scene. Furthermore, the sound effect during the actor's acting is reproduced in real time to give movement, and the sound effect of the performance is maximized by raising or lowering the volume of the sound. In addition, although it is already being played in the background sound, the sound effect was designed as if the wind was passing by giving movement to natural sounds such as wind in real time.
3.6. Experiment Result
Through the experiment, the experimental results of the directionality of implementing the location of the immersive sound source using multiple speaker systems are as follows. First, when the location of the sound source is in the front, the low range has a more pronounced directionality than the high range. Second, as the location of the sound source becomes farther from the front, it is expressed as a plane sound source rather than a point sound source. Third, when the location of the sound source is on the side, the high-pitched range has a more pronounced directionality than the low-pitched range. Fourth, as the position of the sound source moves back from the center of the side, the directional expression decreases. Fifth, when the location of the sound source is on the floor, the low-pitched range can express directionality, but the high-pitched range is expressed by the front floor, so it is difficult to have accurate directionality. Sixth, when the location of the sound source is above the head, it is expressed entirely in the ceiling surface like a plane sound source rather than directionality. As a result, the sound source can be positioned anywhere in space, such as outside the installation location of the speaker or inside the listening seat, but it is difficult to recognize the exact location according to the direction. However, due to the characteristics of the plane wave that allows multiple listeners to perceive the same direction of the sound source, the direction of the sound source is the
same in all seats of the listening seat.
4. Conclusion and Discussion
Immersive sound technology is a technology that can give sound engineers infinite possibilities by providing creativity. The venue, where live performances are performed on stage, is still using stereo sound technology. Now, there is a dire need to apply a next-generation sound system that can satisfy the needs of audiences in concert halls.
This study aims to implement next-generation sound technology in a live performance environment, focusing on production of performances applying immersive sound technology. This study was conducted to effectively deliver immersive sound to the audience using multiple speakers in live performances on the stage of a large theater. In order to apply the immersive sound technology, an acoustic environment that can provide immersion to the audience by studying the process of selecting and installing the most efficient system and producing sound based on the theoretical background were established.
This study attempted to improve the value of
performance as an experiential product by introducing a performance production model applying immersive sound technology. The value of immersive sound was discovered for the usefulness of future performance production and system design. Through the results of this study, 3D sound mixing was illustrated so that it can be helpful in producing performances that implement immersive sound using speakers in the future. The following has the limitations of this study. First, performance has a limitation in that it is difficult to generalize the evaluation of the performance of the applied technology as a result of the study because it has the characteristics of experience in which valuation is performed from a subjective perspective. Second, in the case of the immersive sound produced based on the analysis result after the measurements, the result may differ depending on the environmental characteristics of the space and the genre of music. Third, this study has a limitation in that cost research to implement immersive
sound technology is excluded from the research process. Therefore, the necessity of follow-up research according to the limitations of the research results are suggested. First, the current immersive sound technology continues to develop. Research on various DSP products should be preceded. The understanding of the processor is needed as much as the understanding the exact technology. Second, a large cost is required to implement immersive sound technology with multiple speakers. Since the introduction of next-generation technology requires a great expense, it is best to have a sponsor from a company that is importing the product. A flexible response to product research is required, and a separate study is needed to promote the technology altogether. Research on next-generation technology has several limitations, but it suggests the necessity of the research on how to use the technology to express art because it presents a performance production model applying technology. This study hopes to become a catalyst for research on installation of an immersive sound system in concert halls and establish an acoustic environment that enhances the sense of space and immersion to the audience. In addition, it is expected that various performance contents applied with immersive sound could be produced.
References
[1] Kim, Jinwoong et al 6. How 3D Technology Changes Future. Electronic newspaper. pp.155-160, 2010
[2] Kim, Jung-Hoon, Kwon, Ki-Soo, Kang, Tae-Gyun, Kim, Nam-Su. Current State of the Art and Prospect of User Centric- Realistic Audio Technologies. Journal of
Broadcast Engineering. Volume 19, No. 1, pp. 10-21,
2014.
[3] Lee, Yong-ju, Yoo, Jae-hyun, Jang, Dae-young, Lee, Mi-sook, Lee, Tae-jin. Spatial Audio Technologies for Immersive Media Services. Electronics and
Telecommunications Trends. No. 3, June, pp. 13, 2019.
[4] Go, Eun-ryeong. A Study on the Theatre for the Visually Impaired and Blind. Master's Thesis Korea National University of Arts. Seoul, Korea. 2017.
