AVAD-VR
     Audio/Video Anechoic Database

Principal Contributors

  • Project manager - Brian F.G. Katz
  • Project coordination, Anechoic Recordings, GA Model, Auralizations, - David Thery
  • Assistant recording engineer - Guillaume Leplège - ENS Louis-Lumière
  • Double bass - Antoine Leiser
  • Cello - Noémie Malochet-Breuilh
  • Guitare - Antoine Soler
  • Alto - Eve Nikolaïdis
  • Saxophone - Zephania Lascony
  • Violin - Athanase Nikolaïdis, Baptiste Leblan, Lisa Murcia
Jazz Musicians anechoic room

Project overview

This project relates to the creation of a public database of anechoic audio and 3D-video recordings of several small music ensemble performances. Musical extracts range from baroque to jazz music. This work aims at extending the already available public databases of anechoic stimuli, providing the community with flexible audio-visual content for virtual acoustic simulations. For each piece of music, musicians were first close-mic recorded together to provide an audio performance reference. This recording was followed by individual instrument retake recordings, while listening to the reference recording, to achieve the best audio separation between instruments. In parallel, 3D-video content was recorded for each musician, employing a multiple Kinect~2 RGB-Depth sensors system, allowing for the generation and easy manipulation of 3D point-clouds. Details of the choice of musical pieces, recording procedure, and technical details on the system architecture including post-processing treatments to render the stimuli in immersive audio-visual environments are provided. This work was presented at the 23rd ICA19 conference in Aachen, September 2019.

Citation

  • Thery, D. and Katz, B. F.G., "Anechoic audio and 3D-video content database of small ensemble performances for virtual concerts", ICA conference, Aachen, 2019
  • (url)

Database contents

This database currently includes 6 different musical pieces:

  • Classical pieces
    • Aria, JS Bach (BWV 1068), 1'00"
    • Canon alla ottava, J.S. Bach (BWV 1080/15), 0'54"
    • Summer (4 seasons), A. Vivaldi (RV 315), 3'25"
  • Jazz pieces
    • Minor Swing, D. Reinhardt, 1'15"
    • Don't mean a thing, D. Ellington, 2'00"
    • Si tu vois ma mère, S. Bechet, 1'30"

The recording procedure employed 2 recording sessions: a first one in which all musicians were recorded together, serving as reference for a second session in which each musician was recorded individually.

In parallel to the audio recordings, video recordings were carried out, the final goal being to reconstruct 3D-point clouds avatars of the musicians, playing in a virtual scene.

Please refer to the ICA paper for all associated details.

Anechoic audio and auralization examples

As an example, these anechoic recordings have been used for creating test auralizations for the simulations of two rooms: the Morgan Museum Library, NY (MML), a shoebox shapped chamber music hall, and the amphitheater of the Cité de la Musique, Paris (CM), a fan-shaped chamber music hall. These audio extracts were rendered using simulated 2nd order Ambisonic RIRs and binauralized using the virtual speaker array approach in spat~.
  • Anechoic individual instruments recordings:
  • Auralization of S. Bechet "Si tu vois ma mère", in the Morgan Museum Library (Front and Back positions):
  • Auralization of D. Reinhardt "Minor Swing", in the Amphitheater of the Cité de la Musique (Front and Back positions)

Downloads

Audio anechoic recordings

Sydney Bechet samples Bechet.zip
Django Reinhardt samples Reinhardt.zip
Duke Ellington samples Ellington.zip
Aria (Bach) samples Aria.zip
Canon (The art of the fugue, Bach) samples Canon.zip
4 Seasons (Vivaldi) samples 4Seasons.zip
All audio samples All_Anechoic_Audio.zip

Video recordings - point-clouds

These are the video recordings in .ply format (binary version, check the specifications)
Sydney Bechet samples Bechet_plys.zip
Django Reinhardt samples Reinhardt_plys.zip
Duke Ellington samples Ellington_plys.zip
Aria (Bach) samples Aria_plys.zip
Canon (The art of the fugue, Bach) samples Canon_plys.zip
4 Seasons (Vivaldi) samples 4Seasons_plys.zip
All video samples All_Videos_plys.zip

Orientation data

This archive file contains rotation coordinates of the moving (rotating) instruments, with a Matlab script for processing this data to assist in the inclusion of dynamic source directivity. The orientation data is provided here as text file, including azimuth, elevation, and frame number, to be synchronized with the image.

As detailed in Postma et al, (2017), dynamic directivity can be included in the auralization, using an approach based on the decomposition of the impulse response in 12 beams, angularly equally distributed, and the application of weighting gains on each of these beams based on dynamic instrument orientation. This processing and method has recently been improved with regards to speed and latency, in Katz, Le Conte, & Stitt (2019).

Unity project

This archive file contains a Unity project which will play a jazz extract, running both a Unity scene for the visual, and a Max patch for the audio, triggered by the launch of the scene.

  • VR-Concert-light.zip (attention, 3.6 GB) : Open the main scene, open the Max patch, play the scene. To adapt the played piece, replace each musician path of source images, located in the Resources folder.
Point Cloud in a Unity Scene

Known Issues

  • Due to slight network instabilities, the recording frame rate was not exactly 15 FPS. Hence, some frames have been duplicated to keep audio-video synchronization throughout all pieces.
  • The classical session suffered from Kinect calibration issue, resulting in poorer quality of the 3D-point-cloud. We a currently investigating quality improvement.

Associated / Related work

Conference proceedings

  • Thery, D. and Katz, B. F.G., "Auralization of virtual concerts: a subjective evaluation comparing Ambisonic and binaural rendering", ASA conference, Louisville, 2019: (Abstract only)
  • Thery, D. and Katz, B. F.G., "Anechoic audio and 3D-video content database of small ensemble performances for virtual concerts", ICA conference, Aachen, 2019. (url)
  • Katz, B.; Le Conte, S; Stitt, P.; EVAA: A platform for Experimental Virtual Archeological-Acoustics to study the influence of performance space. Intl Sym on Room Acoustics (ISRA), 2019.

Contact

For any questions, please contact the head of the project: Brian F.G. Katz