Close Show/hide page

Augmented Reality Texture Extraction Experiment

This is an AR-based experiment that enables the user to lift textures from real-world objects in live video and apply them onto 3D objects that are overlayed on top of them. Only box primitives are supported here, but the general idea could be extended to other types of 3D primitives or potentially even more complex objects with some clever image compositing and UV mapping (a-la Photosynth, I guess).


View on Vimeo.
Thumbnail - Click me
Click for live demo

Click the image on the right to run a live version of the experiment, in case you have the patience to suffer thru the quick-and-dirty hotkey-based UI.

Print the AR marker (PDF | PNG) and point your webcam at it.

It’s running Saqoosha’s (feature-incomplete) Alchemy branch of the FLAR Toolkit, along with Papervision3D.

In the future, it would be nice to figure out is how to apply bilinear filtering to the ‘deperspectivized’ textures. I could also add a feature to export the textured 3d objects into a 3D file format (probably OBJ) if there’s any interest.

Conceptually, this piece builds on these two video projection tests, and is a modest implementation of one of the themes from this post of ideas for augmented reality.

23 Responses to “Augmented Reality Texture Extraction Experiment”

  1. Wez Crozier Says:

    Awesome execution and looks like it’s running really fast. Well done Lee!

  2. admin Says:

    Thanks Wez. In the spirit of full disclosure I should say that when I recorded the screencast, I set the application to run all tweens 2x slower, and then baked the video to play back 2x faster than the speed at which it was recorded. :)

  3. Wez Crozier Says:

    Sneaky. ;) How much performance did you gain by using the Alchemy branc of FLAR?

  4. nicoptere Says:

    well done indeed :)
    it reminds me of the early version of carrara.

  5. admin Says:

    @Wez Once I started getting into this project, I didn’t get a chance to compare its performance against the ‘trunk’ version of FLARToolkit. But it seemed a lot more responsive.

    Unfortunately, the current Alchemy branch only seems to support sampling bitmaps at 320×240. In the future, it may be nice to take advantage of the added performance to sample bitmaps for marker detection at a higher resolution.

  6. Bruce Sterling Says:

    Complete Works of Borges. A stellar choice.

  7. TwittLink - Your headlines on Twitter Says:

    [...] zero point nine » Blog Archive » Augmented Reality Texture Extraction Experiment [...]

  8. Marvin Says:

    wow, awesome. keep it up!

  9. Ramon Fritsch Says:

    Nice work man.

    do you make it on flash? very fast!

  10. links for 2009-09-25 | damonky.co.uk -web design and development Says:

    [...] zero point nine » Blog Archive » Augmented Reality Texture Extraction Experiment (tags: augmentedreality ar as3 inspiration flartoolkit textures cool fun) Share and Enjoy: [...]

  11. Rémi.T Says:

    Amazing ! I love it

  12. KJ Says:

    Very nice, incredibly smooth. Can definitely see the potential in this.

  13. Justin Says:

    Amazing! Great idea and execution as always Lee. I like the little Borges cameo too – respect for you literary taste ;)

  14. admin Says:

    Hehe, thanks Justin.

  15. My most important Twitter Messages #3 | der hess Says:

    [...] Very nice Flash Augmented Reality Experiment Idea: Augmented Reality Texture Extraction [...]

  16. Pipo Says:

    Could you please explain how to take a snapshot of a particular area and to use it as BitmapMaterial for example on a Cube?

    How do you got such a good performance? :-)

  17. admin Says:

    Pipo, I’ve greatly simplified the algorithm for the snapshot part which allows it to be applied to any kind of mesh, not just to a plane or cube face, and hope to post a demo and the code for it soon. (I’ve really wanted to do this sooner but have lacked the necessary alignment of time + energy.)

    Because in this demo the camera is assumed to remain stationary for the most part, I’m doing a simple check of the webcam’s activityLevel, and if it’s below a certain value I skip telling FLAR to do a detectMarker, which is where most of the performance hit comes from.

  18. Pipo Says:

    Thanks a lot for your comment! I’m looking forward to your code. Don’t rush with it :-)

  19. Pablo Cabana Says:

    Man, congratulations!
    The idea is just amazing. Keep the good work!!

  20. stoli Says:

    photoshopped

  21. admin Says:

    LOL ;)

  22. Chaco Says:

    great idea please in the meanwhile can you explain more about this. do you mean that if there is no activity you put some kind of delay in the detecMarker function?

    [Because in this demo the camera is assumed to remain stationary for the most part, I’m doing a simple check of the webcam’s activityLevel, and if it’s below a certain value I skip telling FLAR to do a detectMarker, which is where most of the performance hit comes from.]

  23. admin Says:

    Hi Chaco, here’s some pseudocode:

    function onEnterFrame(e:*)
    {
    if (_webcam.activityLevel < 25) return;

    var b:Boolean = _flarDetector.detectMarkerLite(_flarRaster, 80);
    // etc.
    }

Leave a Reply

Powered by WP Hashcash