FLV Encoder with Audio
12 Mar 2011
Demo implementation: Records webcam video and audio to a local file
! See updated FLV Encoder post + code here
A prospective client asked about the possibility of adding audio support to SimpleFlvEncoder. In the end, I was not commissioned for that work, but not before my curiosity had already gotten the better of me...
This class makes it possible to create FLV's in AS3 that contain both video and audio, all on the client-side using the regular browser-based Flash Player. Like with the old version from 2007 (!), video is uncompressed (or rather, just zlib-compressed on a per-frame basis). Audio must be supplied in uncompressed PCM format with either 8 or 16 bits per sample, mono or stereo, and at 11, 22, or 44 KHz. It can also create FLV's that contain only audio or only video.
Here is a quick-and-hand-wavey example of using the class (Please see the comments in the source for more detail.).
var flvEncoder:FlvEncoder = new FlvEncoder(_fps, _duration);
flvEncoder.setVideoProperties(_videoWidth, _videoHeight);
flvEncoder.setAudioProperties(FlvEncoder.SAMPLERATE_44KHZ, _is16Bit, _isStereo);
flvEncoder.begin();
for (var i:int = 0; i < _numFrames; i++)
{
flvEncoder.addFrame(_myBitmaps[i], _myChunksOfAudio[i]);
}
saveOutMyFile( flvEncoder.byteArray );
The old "SimpleFlvEncoder" class, which dealt only with video, was stupidly simple to use. Having to deal with audio ends up being a little more involved. As I said, you have to supply the audio data in PCM format, which basically just means raw, uncompressed audio. You give it the audio data in the addFrame() method. You might gather that data using Sound.extract() or SampleDataEvent, or by dynamically generating your own. Use audioFrameSize to determine how much audio data in bytes is expected per frame.
Supplying too little or too much audio per frame is I believe "legal" per the FLV spec, so the class will take whatever size ByteArray is given to it for audio. An incorrect amount may result either in an unexpected framerate on playback or choppiness in the audio depending on the FLV playback implementation of the client program.
It's beyond the scope of this post to provide a tutorial-like explanation of playing with audio data in Flash, but let me describe what I think are the three most likely use-cases for this class when it comes to audio.
One is where the audio is static (e.g., a soundtrack that accompanies the video). In this case, you could load/embed an uncompressed WAV file into a ByteArray and basically just feed the correct number of bytes from that ByteArray to the addFrame() method without much drama (Uncompressed WAV files are just raw PCM audio data with some metadata at the beginning; you would start copying the bytes from after the end of the metadata). Alternatively, you could use Sound.extract() on an MP3, etc.
Another is where the audio comes from the webcam, which is what I'm doing in the sample implementation above. I'm using a modified version of Thibault Imbert's MicRecorder class to collect the audio data into one big audio ByteArray and then chunking out the pieces of it to FlvEncoder in a loop at the very end, which basically amounts to the same thing as the first scenario.
The last is where the audio is dynamic, presumably accompanying video which is equally dynamic, like for a game or other interactive experience. Unfortunately, it's impossible to record the global sound output from Flash using SampleDataEvent or anything else (though I'd be happy to be proven wrong). Just for the hell of it I tried using SoundMixer.computeSpectrum() on a really quick interval for the task, and was able to get something approximating what was needed, but it was predictably... way not good enough. Anyway, this means you would need to create something like a sound mixing engine, or possibly use some extant AS3 sound compositing library.
The last thing to say on this for the time being is that I tried my hand at Alchemy for the first time and made a version of the encoder that runs about 3x faster, but as a C++ and Alchemy noob, couldn't get rid of a memory leak and then gave up on it for the time being. If you're well versed at this sort of thing please let me know and I will happily send you the code. Otherwise, I will hopefully return to it at some point and post an update.
If the encoder is quick enough and the CPU demands of the dynamically generated video content are low enough, the FLV could be created in real time as the content is being produced, rather than storing all the bitmaps and audio data in memory and doing it all at the end; this might be viable under some limited use cases (i.e., low enough CPU use by Flash + low enough frame rate + small enough video size + fast enough computer).
Last last thing: I've always thought one of the coolest open source AS3 projects would be to use Alchemy with the Xvid or x264 C library to do client-side all-Flash Xvid or h264 video encoding (An "easier" solution would be AIR w/ NativeProcess + ffmpeg, but that kind of ruins the fun.). Anyway, just putting it out there...