PixelBender audio mixer with dynamic track count

Pixel Bender can be a great tool for number crunching, but has a couple of limitations when trying to create an audio mixer.

• The number of inputs is limited when using PixelBender toolkit to compile .pbj files.
• Even if one overcomes the limitation by writing Pixel Bender assembly, the track count is fixed to a predetermined number determined by the number of inputs you have in the .pbj.

In some cases you might want your application to act as a normal audio app where you can add as many tracks as your CPU can handle, but not use more CPU than needed for the current track count.

What we need to accomlish that is a way to dynamically create a shader with the desired number of inputs. James Ward has created pbjAS, a library that enables you to create Pixel Bender shaders at runtime, based on Nicolas Cannasees haXe library.
Also, Tinic Uro has posted Pixel Bender assembly code to create a mixer.

So I set out to use pbjAS to recreate Tinics code with a dynamic number of channels.
The result is this class:

package com.blixtsystems.audio
{
	import flash.display.Shader;
	import flash.display.ShaderJob;
	import flash.utils.ByteArray;
	import pbjAS.ops.OpAdd;
	import pbjAS.ops.OpMul;
	import pbjAS.ops.OpSampleNearest;
	import pbjAS.params.Parameter;
	import pbjAS.params.Texture;
	import pbjAS.PBJ;
	import pbjAS.PBJAssembler;
	import pbjAS.PBJChannel;
	import pbjAS.PBJParam;
	import pbjAS.PBJType;
	import pbjAS.regs.RFloat;
	/**
	 * Shader to mix audio with a dynamic number of channels
	 * @author leo@blixtsystems.com
	 */
	public class MixerShader
	{
		private var _bufferSize:int;

		private var _pbj:PBJ = new PBJ();
		private var _shader:Shader;
		private var _buffer:Vector.<ByteArray> = new Vector.<ByteArray>();

		private var _numTracks:int;

		/**
		 * Constructor
		 * @param	numTracks	track count
		 */
		public function MixerShader(numTracks:int, bufferSize:int=2048)
		{
			_numTracks = numTracks;
			_bufferSize = bufferSize;
		}

		/*-----------------------------------------------------------
		Public methods
		-------------------------------------------------------------*/
		/**
		 * Mix audio
		 * @param	data	ByteArray in which to store the result, probably SampleDataEvent.data
		 */
		public function mix(data:ByteArray):void
		{
			var mixerJob:ShaderJob = new ShaderJob(_shader, data, 1024, _bufferSize/1024);
			mixerJob.start(true);
		}

		/*-----------------------------------------------------------
		Private methods
		-------------------------------------------------------------*/
		private function assembleShader():void
		{
			var channels:Array = [PBJChannel.R, PBJChannel.G];
			var chanStr:String = "rg";
			_pbj.version = 1;
			_pbj.name = "SoundMixer";
			_pbj.parameters =
			[
				new PBJParam
				(
					"_OutCoord",
					new Parameter
					(
						PBJType.TFloat2,
						false,
						new RFloat(	0, channels)
					)
				)

			];
			_pbj.code =
			[
				new OpSampleNearest
				(
					new RFloat(1, channels),
					new RFloat(0, channels),
					0
				),
				new OpMul
				(
					new RFloat(1, channels),
					new RFloat(3, channels)
				)
			];
			var i:int;
			for (i = 0; i < _numTracks; i++)
			{
				_pbj.parameters.push
				(
					new PBJParam
					(
						"track" + i,
						new Texture(2, i)
					)
				);
			}
			for (i = 0; i < _numTracks; i++)
			{
				_pbj.parameters.push
				(
					new PBJParam
					(
						"volume" + i,
						new Parameter
						(
							PBJType.TFloat2,
							false,
							new RFloat(i + 3, channels)
						)
					)
				);
			}
			for (i = 0; i < _numTracks-1; i++)
			{
				_pbj.code.push
				(
					new OpSampleNearest
					(
						new RFloat(2, channels),
						new RFloat(0, channels),
						i+1
					),
					new OpMul
					(
						new RFloat(2, channels),
						new RFloat(i+4, channels)
					),
					new OpAdd
					(
						new RFloat(1, channels),
						new RFloat(2, channels)
					)
				);
			}
			_pbj.parameters.push
			(
				new PBJParam
				(
					"output",
					new Parameter
					(
						PBJType.TFloat2,
						true,
						new RFloat(1, channels)
					)
				)
			);

			var pbjBytes:ByteArray = PBJAssembler.assemble(_pbj);
			_shader = new Shader(pbjBytes);
			_buffer = new Vector.<ByteArray>(_numTracks);

			// initialize the shader inputs
			for (i = 0; i < _numTracks; i++) {
				_buffer[i] = new ByteArray();
				_buffer[i].length = _bufferSize * 4 * 2;
				_shader.data["track" + i]["width"] = 1024;
				_shader.data["track" + i]["height"] = _bufferSize / 1024;
				_shader.data["track" + i]["input"] = _buffer[i];
				_shader.data["volume" + i]["value"] = [1, 1];
			}

		}

		/*-----------------------------------------------------------
		Getters/Setters
		-------------------------------------------------------------*/
		public function get numTracks():int { return _numTracks; }
		public function set numTracks(value:int):void
		{
			// needs to be at least one input, and no point reassembling pbj if track count has not changed
			if (_numTracks < 1 && _numTracks == value) return;

			_numTracks = value;
			assembleShader();
		}

		public function get buffer():Vector.<ByteArray> { return _buffer; }

	}

}

To use it you need to download pjbAS from James Ward and include the swc in your project.

Then in your audio engine do the following:

static public const BUFFER_SIZE:int = 2048;

// number of tracks (minimum 1)
private var _numTracks:int = 1;
// instantiate mixer shader
private var _mixerShader:MixerShader;
_mixerShader = new MixerShader(_numTracks, BUFFER_SIZE);

// here you have your sound objects stored
// adding or removing objects will change number of tracks in the shader
public var sounds:Vector.<Sound> = new Vector.<Sound>();

// SampleDataEvent callback
private function samplesCallback(e:SampleDataEvent):void
{
	_numTracks = sounds.length;

	// update number of track for the shader, causing pbj to recompile
	_mixerShader.numTracks = _numTracks;

	// extract audio data into the shader buffers
	for (var i:int = 0; i <  _numTracks; i++)
	{
		_mixerShader.buffer[i].position = 0;
		sound[i].extract(_mixerShader.buffer[i], BUFFER_SIZE);
		_mixerShader.buffer[i].position = 0;
	}
	// do shader job
	_mixerShader.mix(e.data);
}

According to Tinic Uro, the number of inputs in a shader is limited to 15, so adding more tracks than that will probably not work.

In my tests, playing back eight tracks using pure AS3 will often cause CPU peak usage over 16% of one 2.3 GHz core on my quad-core machine with the debug version of the Flash Player.
If I instead use Pixel Bender the peaks are rarely above 6%. Adding or removing tracks which causes the pbj to recompile, does not cause any noticeable spikes in CPU usage.
So using Pixel Bender will cut CPU usage to around 1/3 on my machine!

Big thanks to Tinic Uro, Nicolas Cannasse and James Ward who made it easy to accomplish!

Share/Bookmark

Related posts:

1. Audio mixing on AIR for android
2. Simple rotating preload icon class
3. Random thoughts on Vectors and samplesCallbackData
4. Tweaking the samplesCallbackEvent loop
5. Base62 encode/decode with AS3 and PHP