Is it not the norm for the agency developer of whatever flavor to be asked make more progress than what’s reasonable based on the current status of a given project at any given point in time?

Thus, we continuously look for ways to make more progress with less information, even while increasing the risk of wasted work and unseemly displays of developer angst. At the very least, such experiences make a person receptive to finding a few tricks that might shave off a few extra minutes in one’s pursuit to meet the latest unreasonable deadline.

That having been said, the potential utility of this little tool should require no further explanation.

It has its origins in this capricious tweet from some time last winter.

- Download (uses Adobe AIR)

- Source (Flex project) ]]> http://www.zeropointnine.com/blog/fpo-image-generator/feed/ 4 http://www.zeropointnine.com/blog/updated-flv-encoder-alchem/ http://www.zeropointnine.com/blog/updated-flv-encoder-alchem/#comments Wed, 30 Mar 2011 14:32:45 +0000 admin http://www.zeropointnine.com/blog/?p=898

This is an update to FLV Encoder which adds an optional Alchemy routine that’s about 3.5x faster, as well as FileStream support for writing directly to a local file using AIR. The library has been architected in such a way that you can use the package while targeting either the browser (no FileStream support) or AIR, and either Flash 9 (no Alchemy support) or Flash 10 – without getting class dependency compiler errors.

I’ve put the package on Github, along with a couple examples (including the one from the last post). The API has changed a little so be sure to also see the example code below. A method updateDurationMetadata() has been added so the video duration does not have to be declared before the video has been created. Also, a bug where the top-most line of pixels was not being written has been fixed.

Realtime encoding demo:

Because of the increased speed of the Alchemy version, it is now viable to encode FLV’s in realtime as the audio and video is being captured, at least within certain limits. Click on the thumbnail above for an online demo that encodes webcam video and audio to a file at 320×240 in real time. If your system is fast enough, you can keep the framerate set to 15FPS with minimal hiccups. The browser-based version must store the entire FLV in memory before saving to disk, but the equivalent AIR version can save its contents directly to a file so that the only limiting factor is disk space. I’m using a dynamic timing and queuing system to keep video and audio in sync which could be the topic of another post.

Updated usage examples:

[A] To encode an FLV to a ByteArray, start by instantiating ByteArrayFlvEncoder. The rest is similar to the last version, but you can now call updateDurationMetadata() when you’re done to update the duration property in the metadata. Finally, call kill() to prepare the object for garbage collection:

[as3]
var baFlvEncoder:ByteArrayFlvEncoder = new ByteArrayFlvEncoder(myFrameRate);

baFlvEncoder.setVideoProperties(myWidth, myHeight, VideoPayloadMakerAlchemy);
// (Omit the 3rd argument to NOT use Alchemy if you’re targeting Flash 9)
baFlvEncoder.setAudioProperties(BaseFlvEncoder.SAMPLERATE_44KHZ, true, false, true);

baFlvEncoder.start();

baFlvEncoder.addFrame(myBitmapData, myAudioByteArray);
baFlvEncoder.addFrame(myBitmapData, myAudioByteArray); // etc.

baFlvEncoder.updateDurationMetadata();

saveOutMyFileUsingFileReference( baFlvEncoder.byteArray );

baFlvEncoder.kill(); // for garbage collection
[/as3]

[B] To encode an FLV directly to a local file (in AIR), instantiate FileStreamFlvEncoder with a File reference, and open up the exposed FileStream, and then close it when you’re all done:

[as3]
var myFile:File = File.documentsDirectory.resolvePath(“video.flv”);
var fsFlvEncoder:FileStreamFlvEncoder = new FileStreamFlvEncoder(myFile, myFrameRate);
fsFlvEncoder.fileStream.openAsync(myFile, FileMode.UPDATE);

fsFlvEncoder.setVideoProperties(myWidth, myHeight, VideoPayloadMakerAlchemy);
fsFlvEncoder.setAudioProperties(BaseFlvEncoder.SAMPLERATE_44KHZ, true, false, true);

fsFlvEncoder.start();

fsFlvEncoder.addFrame(myBitmapData, myAudioByteArray);
fsFlvEncoder.addFrame(myBitmapData, myAudioByteArray); // etc.

fsFlvEncoder.updateDurationMetadata();

fsFlvEncoder.fileStream.close();

fsFlvEncoder.kill();
[/as3]

A new instance of FlvEncoder should be created for every new FLV to be generated.

If you’re using the Alchemy version, the SWC lives in leelib/util/flvEncoder/alchemy, so be sure add that folder to your project library path. The C++ code is there as well.

Note that the current Alchemy video encoding routine is being done synchronously, which means that although it’s much faster than the regular AS3 version, the main Flash thread must still wait for the Alchemy encoding to be done before continuing. An asynchronous version would allow for a real-time encoding scenario that is more flexible and probably with less ‘dropouts’, and would probably prompt the need an internal queuing system. Anyway, this warrants further development.

- Source code (Github)

Licensed under a Creative Commons Attribution 3.0 License.

! Update: See updated FLV Encoder post + code here

A prospective client recently approached me about the possibility of adding audio support to SimpleFlvEncoder. They ultimately decided to go a different direction and so didn’t commission me for that work, but not before my curiosity had already gotten the better of me…

This class makes it possible to create FLV’s in AS3 that contain both video and audio, all on the client-side using the regular browser-based Flash Player. Like with the old version from 2007 (!), video is uncompressed (or rather, just zlib-compressed on a per-frame basis). Audio must be supplied in uncompressed PCM format with either 8 or 16 bits per sample, mono or stereo, and at 11, 22, or 44 KHz. It can also create FLV’s that contain only audio or only video.

Here is a quick-and-hand-wavey example of using the class (Please see the comments in the source for more detail.).

var flvEncoder:FlvEncoder = new FlvEncoder(_fps, _duration);
flvEncoder.setVideoProperties(_videoWidth, _videoHeight);
flvEncoder.setAudioProperties(FlvEncoder.SAMPLERATE_44KHZ, _is16Bit, _isStereo);
flvEncoder.begin();

for (var i:int = 0; i < _numFrames; i++)
{
	flvEncoder.addFrame(_myBitmaps[i], _myChunksOfAudio[i]);
}

saveOutMyFile( flvEncoder.byteArray );

The old "SimpleFlvEncoder" class, which dealt only with video, was stupidly simple to use. Having to deal with audio ends up being a little more involved. As I said, you have to supply the audio data in PCM format, which basically just means raw, uncompressed audio. You give it the audio data in the addFrame() method. You might gather that data using Sound.extract() or SampleDataEvent, or by dynamically generating your own. Use audioFrameSize to determine how much audio data in bytes is expected per frame.

Supplying too little or too much audio per frame is I believe "legal" per the FLV spec, so the class will take whatever size ByteArray is given to it for audio. An incorrect amount may result either in an unexpected framerate on playback or choppiness in the audio depending on the FLV playback implementation of the client program.

It's beyond the scope of this post to provide a tutorial-like explanation of playing with audio data in Flash, but let me describe what I think are the three most likely use-cases for this class when it comes to audio.

One is where the audio is static (e.g., a soundtrack that accompanies the video). In this case, you could load/embed an uncompressed WAV file into a ByteArray and basically just feed the correct number of bytes from that ByteArray to the addFrame() method without much drama (Uncompressed WAV files are just raw PCM audio data with some metadata at the beginning; you would start copying the bytes from after the end of the metadata). Alternatively, you could use Sound.extract() on an MP3, etc.

Another is where the audio comes from the webcam, which is what I'm doing in the sample implementation above. I'm using a modified version of Thibault Imbert's MicRecorder class to collect the audio data into one big audio ByteArray and then chunking out the pieces of it to FlvEncoder in a loop at the very end, which basically amounts to the same thing as the first scenario.

The last is where the audio is dynamic, presumably accompanying video which is equally dynamic, like for a game or other interactive experience. Unfortunately, it's impossible to record the global sound output from Flash using SampleDataEvent or anything else (though I'd be happy to be proven wrong). Just for the hell of it I tried using SoundMixer.computeSpectrum() on a really quick interval for the task, and was able to get something approximating what was needed, but it was predictably... way not good enough. Anyway, this means you would need to create something like a sound mixing engine, or possibly use some extant AS3 sound compositing library.

The last thing to say on this for the time being is that I tried my hand at Alchemy for the first time and made a version of the encoder that runs about 3x faster, but as a C++ and Alchemy noob, couldn't get rid of a memory leak and then gave up on it for the time being. If you're well versed at this sort of thing please let me know and I will happily send you the code. Otherwise, I will hopefully return to it at some point and post an update.

If the encoder is quick enough and the CPU demands of the dynamically generated video content are low enough, the FLV could be created in real time as the content is being produced, rather than storing all the bitmaps and audio data in memory and doing it all at the end; this might be viable under some limited use cases (i.e., low enough CPU use by Flash + low enough frame rate + small enough video size + fast enough computer).

Last last thing: I've always thought one of the coolest open source AS3 projects would be to use Alchemy with the Xvid or x264 C library to do client-side all-Flash Xvid or h264 video encoding (An "easier" solution would be AIR w/ NativeProcess + ffmpeg, but that kind of ruins the fun.). Anyway, just putting it out there...

- Source code for FlvEncoder
- Updated version on Github

Licensed under a Creative Commons Attribution 3.0 License.

I’ve been responsible for the site’s configurator component, which allows the user to create customized maquettes by changing their poses and skin. The “maqet” is then created in physical form through a 3D-printing process and delivered to the user.

Most recently, I’ve been working on the drawing tools that allow the user to paint on a two-dimensional canvas that updates the texture of the 3D model in real-time. Soon to be added is a freehand drawing-to-bezier curve tool.

This is a quick-and-dirty video capture of the configurator in action. But mostly, please feel encouraged to visit the site and try it for yourself.

Download installable AIR file
Requires AIR 2 Beta 2 Runtime +
Windows .NET v3.5

My aim here was to get familiar with AIR 2′s new capability of interacting with external processes’ standard streams.

One useful way of learning how to use a new feature is to proceed directly to trying to abuse it.

To that end, this AIR application takes in a fairly continuous stream of uncompressed binary image data from a native process’ standard output. My hope was to get real-time screen capture updates into AIR at a decent frame rate.

Details and source code after the break–

The native process

For my “native process,” I created a Windows .NET application written in C#. When asked, it takes a screen capture of the desktop and sends the image data through the process’ standard output. For various reasons, I chose to set it up as a Windows form rather than as a plain vanilla console or as a real system service. The relevant C# code is only a few dozen lines long. One interesting function there is bitmapToBytes(), which converts a bitmap into a byte array in the correct format for Flash to be able to apply it directly to a BitmapData without any further (costly) conversion.

The AIR application

On the “client-side,” the Flash AIR application has little to do. It simply sends a command to the native process, telling it to take a screen grab, along with image dimension and offset parameters. Because the client takes in the resulting stream of data in packets whose length should not be assumed to be of any given size, it needs to know where in the data a new image starts and where it ends. So it checks the stream for an arbitrary 4-byte start code as well as a termination code. When the transfer is complete, it just applies the resulting data, collected over time into a ByteArray, using BitmapData.setPixels(). It then requests a new image, and so on.

Performance considerations

On a mid-range quad-core PC using 1280×1024 images (5MB per uncompressed frame), I’m getting about 8 frames per second, or 125ms per frame. The time it takes for the 5MB image to transfer through the standard stream is about 65ms, or 50% of the total time for one iteration; without setting up an honest benchmark test, it’d be unfair and/or impolitic to declare that AIR represents a bottleneck here, so … yeah.

The AIR app eats up about 2/3 of one CPU core using the runtime’s main application thread. The remaining 1/3 represents the amount of time the AIR app has to wait while the native process constructs the screengrab. If the app were re-architected in such a way that the native process always has a new screencap queued up and ready to send – leading to little or no idle time between request and response – the AIR app would easily soak a full CPU core. Since the AIR app is doing little of significance besides reading the process’s standard stream and collecting that info into a byte array, it appears that merely handling large amounts of stdout data is quite costly.

Of course, AIR 2 isn’t even final yet. Nevertheless, I thought these preliminary findings were interesting. (This build targets Beta 2, which is the most recent public beta of AIR 2; compiling and running it against the most recent non-public beta of the SDK and runtime dated April 9, 2010 gave the same results.)

Source files

Setting up a native process for use with AIR 2 is well documented so I didn’t bother commenting on that here. However, the included source is I believe a good starting point for adapting to whatever native process craziness you may be cooking up…

For another implementation of AIR 2 screen capture, see this post by Mihai Corlan, also with source, using Java. There are also a few other related examples out there as well.

Licensed under a Creative Commons Attribution 3.0 License.

Here I’m just expanding upon the ‘texture extraction’ idea posted before. This version maps video onto 3D objects in a much more direct manner by using ray-casting, like the examples in the last several posts.

Controls:

• Mouse rotates object
• Mouse drag moves object
• [SPACE] remaps the texturing of the 3d object
• [ENTER] cycles through different 3D primitives
• [H] shows full list of kludgey hotkey controls

Update with video (8/25/2010):

Although this piece dispenses with the encumbrances of an AR marker or any application-like functionality, I think the method being used here has some potentially interesting uses for an augmented reality-based design application. (Or for more whimsical uses like “upload-your-face” done in 3D). I’ve disabled backface detection, so the imagery gets mapped on all sides of the object rather than just the camera-facing polys, which makes it more visually interesting. Live demo above requires webcam.

Simulates the projection of shadows coming from a spotlight light source onto irregular surfaces. In the supplied example, an animated MD2 model (from Quake 2) is casting a shadow onto the inside of a sphere. Supported parameters include shadow color, alpha, blur, falloff and texture map size, as well as position, rotation and field of view. This evolves out of the projection work posted previously.

Source:

SpotlightShadowCasterManager.as

Usage:

After instantiating the manager –

	_shadowCaster = new SpotlightShadowCasterManager();

– assign one or more DisplayObject3D’s that will block the light from the spotlight:

	_shadowCaster.registerShadowCasterObject(myMesh);

Then register one or more meshes that can have shadows cast upon it with registerShadowCasterObject. As the second argument, you must supply a BitmapMaterial utilized by the mesh on which the manager will draw the shadows. Usually, you’ll want to create a CompositeMaterial, with the BitmapMaterial most likely at the top of the stack. It might look something like this:

	var compositeMaterial = new CompositeMaterial();
	compositeMaterial.addMaterial( new ColorMaterial(0x888888) );
	var mat:BitmapMaterial = new BitmapMaterial( new BitmapData(1,1) );
	// .. the bitmapdata will get overwritten,
	//    but must contain _something_
	var sphere:TriangleMesh3D = new Sphere(compositeMaterial, 1000);

	_shadowCaster.registerShadowReceiverObject(sphere, mat);

To update the shadow textures, call

	_shadowCaster.update();

– probably in your onEnterFrame or onRenderTick function.

That’s enough to get going. From there, it’s just a matter of adjusting the projector’s position and orientation in relation to the objects in your scene with the properties projectorPosition, projectorPitch, and projectorRoll. Check the source for the various configurable properties like blur, alpha, color, etc.

Limitations, bugs:

- As described above, objects must be manually assigned to be either a ‘caster’ or a ‘receiver’. These roles aren’t determined dynamically by the class itself.

- Only one level of shadows is supported. Adding an arbitrary number wouldn’t be difficult, just cumbersome to manage under the current system, and increasingly expensive.

- Unfortunately, a ‘shadow receiver’ object can’t have its own texture in addition to the one used to cast shadows onto, at least not in a way that looks decent. The reason is that a 3d object in Papervision3D has only one set of UV texture data and the shadow texture overwrites this with every update.

- There’s undoubtedly a less expensive way to do the raycasting/projection in the main math routine in the private function updateMesh. If you’re adept at this kind of thing, please send your improvements!

- Ugly and obvious artifacts show up on the ‘projectable’ textures when the spotlight is rotated 90 degrees away them. (You may have already noticed this in the previous two projection experiment posts). Again, I’d love any solutions anyone has for this.

Licensed under a Creative Commons Attribution 3.0 License.

This version handles arbitrary rotations, field of view / focal length, and uses correct math; the previous example faked it somewhat, doing everything basically with 2D math (but why admit to that? :). I found it twice as difficult getting to this point compared to the version posted a few days ago. It turns out that when you want to rotate more than 90 degrees around the x-axis, it’s good to make friends with quaternions, if only superficially.

And it’s interactive this time. By the way, is it not a royal pain to design an interface to control more than two axes in Flash for the web?

There’s still a lot that could be done right / better with this: backface culling (already tried and failed); frustrum culling; per-triangle alpha based on distance and angle to simulate light intensity; per-triangle focus/blur based on distance, maybe (but probably not); a more straightforward way to implement video; shadows… Basically, anything short of building a renderer-inside-a-renderer.

Bonus points for identifying the band to whom the set list pictured in the third image belongs.

Sorry, still no source.

  ]]> http://www.zeropointnine.com/blog/video-projector-effect-advanced/feed/ 5 http://www.zeropointnine.com/blog/video-projector-effect/ http://www.zeropointnine.com/blog/video-projector-effect/#comments Wed, 27 May 2009 02:25:50 +0000 Lee http://www.zeropointnine.com/blog/?p=324

My general goal for this was to approximate the look of a video projection hitting a ‘non-planar surface’ or otherwise intersecting objects in a 3d scene.

The underlying logic wasn’t too painful. The UV coordinates of each vertex are all that need to be solved for. There’s a linear relationship between the position and orientation of the ‘projector’, and the world space position and UV coordinates of each vertex in the scene — requiring just algebra, which is nice, because that’s about all the math I know. JiglibFlash is being used for the physics action.

In its current implementation, the projector is limited to looking straight down the z-axis. I have a feeling that the most elegant approach for all this would be to extend Camera3D or to piggyback on top of the plumbing of the Papervision framework in some other way (apologies for the mixed metaphor). Should I develop the ‘Projection Manager’ to handle arbitrary projector orientation, I’ll post the source along with an updated example.