zero point nine

This is an AR-based experiment that enables the user to lift textures from real-world objects in live video and apply them onto 3D objects that are overlayed on top of them. Only box primitives are supported here, but the general idea could be extended to other types of 3D primitives or potentially even more complex objects with some clever image compositing and UV mapping (a-la Photosynth, I guess).

Click for live demo

Click the image on the right to run a live version of the experiment, in case you have the patience to suffer thru the quick-and-dirty hotkey-based UI.

Print the AR marker (PDF | PNG) and point your webcam at it.

It’s running Saqoosha’s (feature-incomplete) Alchemy branch of the FLAR Toolkit, along with Papervision3D.

Simulates the projection of shadows coming from a spotlight light source onto irregular surfaces. In the supplied example, an animated MD2 model (from Quake 2) is casting a shadow onto the inside of a sphere. Supported parameters include shadow color, alpha, blur, falloff and texture map size, as well as position, rotation and field of view. This evolves out of the projection work posted previously.

Source:

SpotlightShadowCasterManager.as

Usage:

After instantiating the manager –

	_shadowCaster = new SpotlightShadowCasterManager();

– assign one or more DisplayObject3D’s that will block the light from the spotlight:

	_shadowCaster.registerShadowCasterObject(myMesh);

Then register one or more meshes that can have shadows cast upon it with registerShadowCasterObject. As the second argument, you must supply a BitmapMaterial utilized by the mesh on which the manager will draw the shadows. Usually, you’ll want to create a CompositeMaterial, with the BitmapMaterial most likely at the top of the stack. It might look something like this:

	var compositeMaterial = new CompositeMaterial();
	compositeMaterial.addMaterial( new ColorMaterial(0x888888) );
	var mat:BitmapMaterial = new BitmapMaterial( new BitmapData(1,1) );
	// .. the bitmapdata will get overwritten,
	//    but must contain _something_
	var sphere:TriangleMesh3D = new Sphere(compositeMaterial, 1000);

	_shadowCaster.registerShadowReceiverObject(sphere, mat);

To update the shadow textures, call

	_shadowCaster.update();

– probably in your onEnterFrame or onRenderTick function.

That’s enough to get going. From there, it’s just a matter of adjusting the projector’s position and orientation in relation to the objects in your scene with the properties projectorPosition, projectorPitch, and projectorRoll. Check the source for the various configurable properties like blur, alpha, color, etc.

Limitations, bugs:

(more…)

My general goal for this was to approximate the look of a video projection hitting a ‘non-planar surface’ or otherwise intersecting objects in a 3d scene.

The underlying logic wasn’t too painful. The UV coordinates of each vertex are all that need to be solved for. There’s a linear relationship between the position and orientation of the ‘projector’, and the world space position and UV coordinates of each vertex in the scene — requiring just algebra, which is nice, because that’s about all the math I know. JiglibFlash is being used for the physics action.

In its current implementation, the projector is limited to looking straight down the z-axis. I have a feeling that the most elegant approach for all this would be to extend Camera3D or to piggyback on top of the plumbing of the Papervision framework in some other way (hint, hint)(apologies for the mixed metaphor). Should I develop the ‘Projection Manager’ to handle arbitrary projector orientation, I’ll post the source along with an updated example.

Or at least, that’s the best easy analogy to describe it.

Gist: The color of each triangle in the mesh is obtained by averaging the pixel colors of the underlying image at the screen positions under each vertex as well as at the triangles’ center point. Some FlatShader-like lighting is used to add a little more three-dimensionality. Finally, a BlurFilter is applied to the source image to keep the transitions between colors from feeling too abrupt.

Source code: PvColorFacets.as

Licensed under a Creative Commons Attribution 3.0 License.

Now that I’ve finally allowed myself to look at couple online examples of so-called augmented reality, I’ve become quite excited by its possibilities. (Eg, why the hell did I not see this sooner?) But since I have a semi-unhealthy desire to act on every passing interesting idea that comes to mind, I figure I can at least partially purge myself of feeling torn between executing 36 different ideas I don’t have time to and shouldn’t act on by publishing the following notes I’ve compiled over the last couple days… Nonetheless, I’m sure I’ll steal some (or more likely, a lot of) time over the next period exploring one or two of these ideas.

A green-screen/chroma key effect utilizing Pixel Bender. With this version, the live video is compared against a reference image rather than a static color. Additionally, the alpha of the output is graduated based on the color difference between the base image and the incoming video (rather than just being either ‘on’ or ‘off’). Both of these differences may or may not be good for practical use, but can create interesting imagery.

Use the sliders to adjust the cutoff/falloff curve for the alpha. Click “Reset base image” to set the base image with which to compare any further changes in the video against. You can also choose your own background image (click “Browse…” at the bottom). Works best against a plain background and presumably best of all with proper lighting against a green background…

When I wrote the image processing routine in pure Actionscript, it ran like a slideshow. The Pixel Bender version on a 4-core system runs about 30x faster.

Pixel Bender kernel: GreenScreenEffect.pbk

Licensed under a Creative Commons Attribution 3.0 License.

view at full size >

How it would look if you took a webcam or camera, and captured several frames of the same scene in quick succession, and then essentially averaged them out and composited them into one? My guess was that the resulting image would retain most of its sharpness but lose most of the ‘noise’. I made this quick mini-app with that thought in mind.

Here is an example of my results (before and after). The images were captured with a Logitech Quickcam Pro 9000 at a resolution of 1600×1200 (averaged from 128 (!) still images), and were not scaled or retouched.

As a side benefit, if the camera — or something in the foreground of the scene — moves during the image capturing process, you can get photography-like effects that look like a long exposure…

A PNG48 encoder class…

(more…)

view at full size >

Sooner than expected (thanks no doubt to my first snow storm in two years giving me all the excuse I needed to stay in all day), this version covers bullet-point 2 of the previous post:

I’m using Adobe’s Kuler service to generate the color scheme, which I find quite exciting. Because it saves this particular demo from a world of red and black — which are basically the only colors I use when left to my own devices. (Press [C] to cycle through colors) Many thanks to Aden Forshaw for the AS3/Kuler example.

Finally worth noting is that the ‘physics’ have improved a little. I was previously using vertex count as a proxy for mass. Using the volume of the axis-aligned bounding box ends up being a better way of faking it.

A demonstration of splitting a mesh in two along an arbitrary plane. With some faux-physics for added value. I was thinking about the ‘laser room scene’ in Resident Evil, LOL.

I’ve included the source code of a minimal example showing how to cut up the faces of a mesh along a plane:

PlaneSplitMinimalExample.as | compiled swf

This is the foundation for everything in this demo, and for the following ideas which I think would be nice to see executed in Flash–

Make sure to try resetting the scene, pausing (with spacebar), clicking through a mesh several times, and then unpausing..

Oh yeah, and the usual disclaimer/apologies about how much this demo hates your CPU, and how unoptimized it is (e.g., right now every face in the scene is tested against the plane with every click (ouch!)). Built with the most recent revision of Papervision 3D.

Also, apologies for the long hiatus. I’ve been busy moving to the NYC. And working a lot. But y’know, whatever..

UPDATE: John Lindquist has kindly pointed out there’s already a helper function in the Papervision framework for cutting meshes with planes! – MeshUtil.cutTriangleMesh().

Licensed under a Creative Commons Attribution 3.0 License.