Unity/RTcmix (uRTcmix)

November, 2019

Sound localization with the RTcmix LOCALIZE() instrument

rtcmix.org documentation for LOCALIZE()
LOCALIZE() video tutorial

initial setup

The LOCALIZE() RTcmix instrument uses a sound 'ray-tracing' approach combined with a simple HRTF model to position a sound source in virtual space relative to a listener. In Unity, the 'sound listener' for a given scene is usually the Main Camera. We will use that as our destination for sound emanating from a source.

We need to add the C# script getMyTransform.cs to the Main Camera. This will allow us to get the proper set of transform coordinates to use with the LOCALIZE() instrument. Add this C# script to your Assetts and drag it onto the Main Camera.

localize a sound source

Instead of having the LOCALIZE() instrument operate on the Main Camera object (keeping all the incoming audio streams separate for individual localization would be very difficult), we will instead apply the localization delays and amplitude calculations at the sound source. To do this, we will need to find out from the Main Camera what the sound source position is relative to the camera so that the correct parameters can be used. This is what the "getMyTransform.cs" script does.

We declare a class variable to reference the getMyTransform component we have added to the Main Camera:

public class thesoundsource : MonoBehaviour {
	int objno = 0; // set this to the RTcmix instance you are using for this script
	rtcmixmain RTcmix;
	private bool did_start = false;

	getMyTransform myrelativetransform; // to get the position of this object relative to the Main Camera

We then locate that component on the Main Camera:

	private void Awake()
	{
		myrelativetransform = Camera.main.GetComponent<getMyTransform>();
	}

Next we connect the output(s) of any sound-generating instruments attached to our sound source object to the LOCALIZE() instrument, and we start it running with PFields attached to the source coordinate parameters (this is usually done in the Start() method):

		string score = "srcX = makeconnection(\"inlet\", 1, 10) " +
			"srcY = makeconnection(\"inlet\", 2, 10) " +
			"srcZ = makeconnection(\"inlet\", 3, 10) " +
			"LOCALIZE(0, 0, 9999, 10, srcX,srcY, srcZ, 0,0,0, 1.0, 3.0, 0, 1, 2, 0.0001, 100) ";

		RTcmix.SendScore(score, objno);

All we then need to do is to update the 'srcX/srcY/srcZ' PFields from the location of the sound source, relative to the Main Camera sound listener. We can do this in the Update() method, relying upon the getTransform() script of the getMyTransform() component we added to the Main Camera:

	void Update()
	{
		Vector3 srcpos = myrelativetransform.getTransform(gameObject); // this source relative to the 'observer' object

		// note swap of y and z axes
		RTcmix.setpfieldRTcmix(1, srcpos.x, objno);
		RTcmix.setpfieldRTcmix(2, srcpos.z, objno);
		RTcmix.setpfieldRTcmix(3, srcpos.y, objno);
	}

The swap of the y and z axes are because of the Unity convention of using the y-axis as the vertical axis instead of the z axis.

Of course this setup of the LOCALIZE() instrument assumes that the sound source has been configured to pass through the LOCALIZE() instrument. This is done by using the rtinput("AUDIO") scorefile command in a chain of sound-producing game objects, or by using the bus_config() system:

bus_config("SOUNDSOURCEINSTRUMENT", "aux 0-1 out")
bus_config("LOCALIZE", "aux 0-1 in", "out 0-1")
etc...

One warning: if a sound source is moving quickly, a rapid clickihg or 'zipper' noise might occur. This is because the source position is being updated at the slow frame rate, causing visually imperceptible 'jumps' in the position for each frame rendered. However, these 'jumps' can introduce discontinuities in the audio signal, resulting in the noise. This can be minimized or eliminated by using the "smooth" filter type of the RTcxmix makefilter() command. makefilter() with a "smooth" filter type interpolates values coming through a PField, acting as a low-pass filter to smooth out the discontinuities. An example of how this is set up:

		string score = "insrcX = makeconnection(\"inlet\", 1, 10) " +
			"insrcY = makeconnection(\"inlet\", 2, 10) " +
			"insrcZ = makeconnection(\"inlet\", 3, 10) " +
			"srcX = makefilter(insrcX \"smooth\", 50): +
			"srcY = makefilter(insrcY, \"smooth\", 50): +
			"srcZ = makefilter(insrcZ, \"smooth\", 50): +
			"LOCALIZE(0, 0, 9999, 10, srcX,srcY, srcZ, 0,0,0, 1.0, 3.0, 0, 1, 2, 0.0001, 100) ";

		RTcmix.SendScore(score, objno);

LOCALIZE() instrument parameters

Parameters with an asterisk(*) are dynamic PFIelds:

   p[0] = output skip time
   p[1] = input skip time
   p[2] = duration
   *p[3] = overall amp
   *p[4] = source x
   *p[5] = source y
   *p[6] = source z
   *p[7] = dest x
   *p[8] = dest y
   *p[9] = dest z
   *p[10] = headwidth (units)
   p[11] = feet/unit scaler
   p[12] = input channel
   p[13] = behind head filter on/off (simple HRTF)
   p[14] = amp/distance calculation flag
      0: no amp/distance
      1: linear amp/distance
      2: inverse square amp/distance (physically 'correct')
   p[15] = minimum amp/distance multiplier (the smallest amp calculated
   p[16] = maximum distance (for linear amp/distance scaling, p14 == 1)

Often a large 'overall amp' (p[3]) value will need to be used, especially if the inverse-square distance calculation (p14 = 2) is used. This is essentially how sound works in the Real World, but we have no reflection calculations from nearby surfaces, so the amplitude falls very rapidly. It's as if everything was happening in an anechoic chamber. Altering the overall amp multiplier can compensate for this. Use values that are necessary to achieve the results you want.