Is it not the norm for the agency developer of whatever flavor to be asked make more progress than what’s reasonable based on the current status of a given project at any given point in time?

Thus, we continuously look for ways to make more progress with less information, even while increasing the risk of wasted work and unseemly displays of developer angst. At the very least, such experiences make a person receptive to finding a few tricks that might shave off a few extra minutes in one’s pursuit to meet the latest unreasonable deadline.

That having been said, the potential utility of this little tool should require no further explanation.

It has its origins in this capricious tweet from some time last winter.

- Download (uses Adobe AIR)

- Source (Flex project) ]]> http://www.zeropointnine.com/blog/fpo-image-generator/feed/ 5 http://www.zeropointnine.com/blog/maqet-3d-gallery-app-for-android/ http://www.zeropointnine.com/blog/maqet-3d-gallery-app-for-android/#comments Tue, 18 Oct 2011 22:03:08 +0000 admin http://www.zeropointnine.com/blog/?p=999

I just completed an Android app which is a simple 3D gallery of user-created designs for maqet.com.

It was fun to be able to apply previous work – the MAQET configurator component done in Flash – to a completely separate development environment (native Android).

Happily, I was able to use min3D for the OpenGL action, which allows me the opportunity to mention that min3D has been nominated for Packt Publishing’s 2011 Open Source Awards.


- Android Market link

OK, that’s a lie. First I did some trace-to-the-console stuff, then I did the obligatory “paint with your finger” type deal, and then spent a relatively short but intensive period on this as a more finished piece. It was meant as an exercise in learning some iOS API, coding patterns and idioms. I also had to learn how to use a Mac seriously for the first time, which apparently I’m not ashamed to admit saying on the record.

I recycled the maze algorithm posted previously, and kept that part in C++. Then I combined it with live camera input, which I guess is my habit. I got the sense early on that the end result wouldn’t necessarily be spectacular but forged ahead anyway for the reasons mentioned above, and tried to make it as finished as possible.

I haven’t submitted this little toy to the App Store™ for various reasons, some of which are arguably irrational, and it’s possible I should reconsider. Should that come to pass, I will be sure to post an update.

A few more details…

User-selectable parameters determine which pixels get erased from the image. This can be based on brightness, or by the apparent red-ness, green-ness, or blue-ness of the image. The calculations for the latter look like this, which I think worked out quite well. If we’re testing for the “redness” of a pixel…

int redness = (redChannel - greenChannel) + (redChannel - blueChannel);
if (redness < someThresholdValue) eraseMyPixel();


Redness can be a good criteria for filtering for exposed skin, depending on skin tone. Greenness can be a good filter for typical indoor settings because it tends to show up less often as red or blue (unless the walls were painted green, which would in any case lead to another interesting use case (green screen...)). You can see in the video where I'm waving around a green folder how well the green gets isolated.

So anyway, the image is scaled to the same dimensions of the maze so a 1-to-1 comparison can be made between the pixels and the squares of the maze. Blocks of the maze are removed based on whether their corresponding pixels have been erased, and the adjacent vertical or horizontal segment/s are filled in for aesthetic effect. At that point, the maze is, unfortunately, no longer solvable.

And the video feed can also be switched between the front and rear cameras of the iPad 2 at runtime, so that's fun.

Capturing of camera data in iOS is done on a different thread, so concurrency issues turned out to be a big deal, particularly since the program runs in a continuous loop. I tried to do as much of the image processing as possible on that secondary thread to take advantage of the dual core-ness of the iPad 2, but could never quite make it work out. In the end, I had to hand off the imagedata off to the UI thread to do the rest of the work, but at least learned the value of the NSObject method performSelectorOnMainThread..

Various effects with variations can be triggered and combined in real time on both saved and live video:

An ostentatious particle emitter effect:

Explosions which can be triggered on demand or based on sudden changes to the bounding box:

“Alpha trails” (with more hands)…

Strobe, with quick time shifting effect:

The priority for this has been to develop my up-til-now shallow knowledge of C++, focusing on code design and performance, rather than the exploration of novel uses of the Kinect as an input device (or so I tell myself).

Given more time, I’d like to do more with the visual effects, look into shaders, write a polygon-based renderer (rather than simple points), and find excuses to do more with multi-threading (currently the loading routine runs on a separate thread, and the ‘video capture’ routine runs on three). And find a clever use for OpenCV to do more than what I’m doing now with just a bounding box. And play with MIDI to trigger video clips, and map various parameters DAW-style. And then maybe work on a real user interface, possibly broken out in a separate window, using Flash or what have you.

I believe the core recording and playback system and my implementation of a “Point Cloud Video” file format are logical candidates for a public post on GitHub, but will entail some rationalization and massaging of code.

This is an update to FLV Encoder which adds an optional Alchemy routine that’s about 3.5x faster, as well as FileStream support for writing directly to a local file using AIR. The library has been architected in such a way that you can use the package while targeting either the browser (no FileStream support) or AIR, and either Flash 9 (no Alchemy support) or Flash 10 – without getting class dependency compiler errors.

I’ve put the package on Github, along with a couple examples (including the one from the last post). The API has changed a little so be sure to also see the example code below. A method updateDurationMetadata() has been added so the video duration does not have to be declared before the video has been created. Also, a bug where the top-most line of pixels was not being written has been fixed.

Realtime encoding demo:

Because of the increased speed of the Alchemy version, it is now viable to encode FLV’s in realtime as the audio and video is being captured, at least within certain limits. Click on the thumbnail above for an online demo that encodes webcam video and audio to a file at 320×240 in real time. If your system is fast enough, you can keep the framerate set to 15FPS with minimal hiccups. The browser-based version must store the entire FLV in memory before saving to disk, but the equivalent AIR version can save its contents directly to a file so that the only limiting factor is disk space. I’m using a dynamic timing and queuing system to keep video and audio in sync which could be the topic of another post.

Updated usage examples:

[A] To encode an FLV to a ByteArray, start by instantiating ByteArrayFlvEncoder. The rest is similar to the last version, but you can now call updateDurationMetadata() when you’re done to update the duration property in the metadata. Finally, call kill() to prepare the object for garbage collection:

[as3]
var baFlvEncoder:ByteArrayFlvEncoder = new ByteArrayFlvEncoder(myFrameRate);

baFlvEncoder.setVideoProperties(myWidth, myHeight, VideoPayloadMakerAlchemy);
// (Omit the 3rd argument to NOT use Alchemy if you’re targeting Flash 9)
baFlvEncoder.setAudioProperties(BaseFlvEncoder.SAMPLERATE_44KHZ, true, false, true);

baFlvEncoder.start();

baFlvEncoder.addFrame(myBitmapData, myAudioByteArray);
baFlvEncoder.addFrame(myBitmapData, myAudioByteArray); // etc.

baFlvEncoder.updateDurationMetadata();

saveOutMyFileUsingFileReference( baFlvEncoder.byteArray );

baFlvEncoder.kill(); // for garbage collection
[/as3]

[B] To encode an FLV directly to a local file (in AIR), instantiate FileStreamFlvEncoder with a File reference, and open up the exposed FileStream, and then close it when you’re all done:

[as3]
var myFile:File = File.documentsDirectory.resolvePath(“video.flv”);
var fsFlvEncoder:FileStreamFlvEncoder = new FileStreamFlvEncoder(myFile, myFrameRate);
fsFlvEncoder.fileStream.openAsync(myFile, FileMode.UPDATE);

fsFlvEncoder.setVideoProperties(myWidth, myHeight, VideoPayloadMakerAlchemy);
fsFlvEncoder.setAudioProperties(BaseFlvEncoder.SAMPLERATE_44KHZ, true, false, true);

fsFlvEncoder.start();

fsFlvEncoder.addFrame(myBitmapData, myAudioByteArray);
fsFlvEncoder.addFrame(myBitmapData, myAudioByteArray); // etc.

fsFlvEncoder.updateDurationMetadata();

fsFlvEncoder.fileStream.close();

fsFlvEncoder.kill();
[/as3]

A new instance of FlvEncoder should be created for every new FLV to be generated.

If you’re using the Alchemy version, the SWC lives in leelib/util/flvEncoder/alchemy, so be sure add that folder to your project library path. The C++ code is there as well.

Note that the current Alchemy video encoding routine is being done synchronously, which means that although it’s much faster than the regular AS3 version, the main Flash thread must still wait for the Alchemy encoding to be done before continuing. An asynchronous version would allow for a real-time encoding scenario that is more flexible and probably with less ‘dropouts’, and would probably prompt the need an internal queuing system. Anyway, this warrants further development.

- Source code (Github)

Licensed under a Creative Commons Attribution 3.0 License.

! Update: See updated FLV Encoder post + code here

A prospective client recently approached me about the possibility of adding audio support to SimpleFlvEncoder. They ultimately decided to go a different direction and so didn’t commission me for that work, but not before my curiosity had already gotten the better of me…

This class makes it possible to create FLV’s in AS3 that contain both video and audio, all on the client-side using the regular browser-based Flash Player. Like with the old version from 2007 (!), video is uncompressed (or rather, just zlib-compressed on a per-frame basis). Audio must be supplied in uncompressed PCM format with either 8 or 16 bits per sample, mono or stereo, and at 11, 22, or 44 KHz. It can also create FLV’s that contain only audio or only video.

Here is a quick-and-hand-wavey example of using the class (Please see the comments in the source for more detail.).

var flvEncoder:FlvEncoder = new FlvEncoder(_fps, _duration);
flvEncoder.setVideoProperties(_videoWidth, _videoHeight);
flvEncoder.setAudioProperties(FlvEncoder.SAMPLERATE_44KHZ, _is16Bit, _isStereo);
flvEncoder.begin();

for (var i:int = 0; i < _numFrames; i++)
{
	flvEncoder.addFrame(_myBitmaps[i], _myChunksOfAudio[i]);
}

saveOutMyFile( flvEncoder.byteArray );

The old "SimpleFlvEncoder" class, which dealt only with video, was stupidly simple to use. Having to deal with audio ends up being a little more involved. As I said, you have to supply the audio data in PCM format, which basically just means raw, uncompressed audio. You give it the audio data in the addFrame() method. You might gather that data using Sound.extract() or SampleDataEvent, or by dynamically generating your own. Use audioFrameSize to determine how much audio data in bytes is expected per frame.

Supplying too little or too much audio per frame is I believe "legal" per the FLV spec, so the class will take whatever size ByteArray is given to it for audio. An incorrect amount may result either in an unexpected framerate on playback or choppiness in the audio depending on the FLV playback implementation of the client program.

It's beyond the scope of this post to provide a tutorial-like explanation of playing with audio data in Flash, but let me describe what I think are the three most likely use-cases for this class when it comes to audio.

One is where the audio is static (e.g., a soundtrack that accompanies the video). In this case, you could load/embed an uncompressed WAV file into a ByteArray and basically just feed the correct number of bytes from that ByteArray to the addFrame() method without much drama (Uncompressed WAV files are just raw PCM audio data with some metadata at the beginning; you would start copying the bytes from after the end of the metadata). Alternatively, you could use Sound.extract() on an MP3, etc.

Another is where the audio comes from the webcam, which is what I'm doing in the sample implementation above. I'm using a modified version of Thibault Imbert's MicRecorder class to collect the audio data into one big audio ByteArray and then chunking out the pieces of it to FlvEncoder in a loop at the very end, which basically amounts to the same thing as the first scenario.

The last is where the audio is dynamic, presumably accompanying video which is equally dynamic, like for a game or other interactive experience. Unfortunately, it's impossible to record the global sound output from Flash using SampleDataEvent or anything else (though I'd be happy to be proven wrong). Just for the hell of it I tried using SoundMixer.computeSpectrum() on a really quick interval for the task, and was able to get something approximating what was needed, but it was predictably... way not good enough. Anyway, this means you would need to create something like a sound mixing engine, or possibly use some extant AS3 sound compositing library.

The last thing to say on this for the time being is that I tried my hand at Alchemy for the first time and made a version of the encoder that runs about 3x faster, but as a C++ and Alchemy noob, couldn't get rid of a memory leak and then gave up on it for the time being. If you're well versed at this sort of thing please let me know and I will happily send you the code. Otherwise, I will hopefully return to it at some point and post an update.

If the encoder is quick enough and the CPU demands of the dynamically generated video content are low enough, the FLV could be created in real time as the content is being produced, rather than storing all the bitmaps and audio data in memory and doing it all at the end; this might be viable under some limited use cases (i.e., low enough CPU use by Flash + low enough frame rate + small enough video size + fast enough computer).

Last last thing: I've always thought one of the coolest open source AS3 projects would be to use Alchemy with the Xvid or x264 C library to do client-side all-Flash Xvid or h264 video encoding (An "easier" solution would be AIR w/ NativeProcess + ffmpeg, but that kind of ruins the fun.). Anyway, just putting it out there...

- Source code for FlvEncoder
- Updated version on Github

Licensed under a Creative Commons Attribution 3.0 License.

I started with a description of the algorithm found on Wikipedia, worked it out in Processing (with 2D, 3D, and isometric views added as a bonus), and then ported it to C++ using openFrameworks for the basic display action.

Presumably it could be used as the start of a game-like concept, if so inclined.

- Processing (source + Win/Mac executables)
- C++/oF (source + Windows executable)

I’ve been responsible for the site’s configurator component, which allows the user to create customized maquettes by changing their poses and skin. The “maqet” is then created in physical form through a 3D-printing process and delivered to the user.

Most recently, I’ve been working on the drawing tools that allow the user to paint on a two-dimensional canvas that updates the texture of the 3D model in real-time. Soon to be added is a freehand drawing-to-bezier curve tool.

This is a quick-and-dirty video capture of the configurator in action. But mostly, please feel encouraged to visit the site and try it for yourself.

I’m looking forward to posting new and hopefully interesting things here as I progress through the program, the time during which I aim to broaden the scope of my personal sandbox and make experimentation a full-time vocation. But hopefully without losing any of the snark.

Video updated with most recent version running on Honeycomb, 6/2011

It was, to be frank, a gigantic pain in the ass coding the motion for this, since there seems to be no Tweener-like class written in Java (Java, not processing…). Anyone know of one?

Built using min3D.

free version

almost-free version :T