It’s been almost a year exactly since my last update. I guess I’ve been pretty preoccupied. I wanted to do a post about an age old problem very often encountered in Flash – compositing video so that overlaying elements sync perfectly with a video. In essence, getting the true current frame of an FLV or F4V. Specifically the problem I want to talk about tackling is having an image be composited as if it were in the video. I think a great example of this is a site we (as in Firstborn, not me) did for 5 React a year or so ago. There are a couple of solutions to solve this problem that work almost perfectly 99% of the time. In general, they usually involve an XML file that contains corner points for each frame of how an image should be transformed to match the video. For live video (no CG) tracking is set up in After Effects and then the cue points are exported. While After Effects doesn’t have a native export cue points as XML option, the keyframe data can be copied & pasted out as plain text, and then you can just run a simple RegEx to convert it to XML. This is all pretty simple and isn’t the problem, the issue lies with not being able to get the exact frame that a video is displaying.
Just a quick note that I will be
regularly contributing to Creativity Online’s CATScan blog. It’s a collection of post from various people about digital media and advertising. I’ll be posting one article per month along with Dave Snyder, who is one of the Creative Directors at Firstborn. My first article is titled ‘It Doesn’t Hurt to Twitter Up Close and Personal’ and is about how businesses can use twitter to intimately reach their audience. My first article is linked here: It Doesn’t Hurt to Twitter Up Close and Personal.
Just a quick update here, we recently relaunched the new Doritos.com [edit: no longer online], my latest project. I was lead developer on this project with much help from Lucas Motta and Michael Roushey on the flash forefront, and as always Francis Turmel as our technical Sensei. The project was done with the world renowned ad agency Goodby, Silverstein & Partners. We put a lot of hours into this site, but I think it was worth it. My job as lead developer was doing the overall structure and framework of the site. I also developed some helpful debugging utils along the way, which I hope to post on here rather soon.
Recently I was working on preloader for a project and needed to use a tweening engine. (We have an internally developed one at Firstborn that is based off of the syntax of Tweener). I wanted to see if I could write my own that would be super small. I wasn’t as concerned with performance – meaning being able to tween thousands of objects at once, a highly unlikely scenario anyways. I wanted to keep compiled file size down, as well as number of classes. So I created a NanoTween class that is a super simple tweening utility. When compiled on it’s own, it only consist of about 1.3 Kb.
I probably should have mentioned this way earlier, but I’ll be speaking along with Mathieu Badimon at OFFF Paris on behalf of Firstborn. Our presentation will be on Friday, June 25th at 12:30. I have to say, this is quite an honor to speak at this conference. If you’re interested, there should be more information on the OFFF Paris 2010 website.
So I realize this is pretty last minute, but I’ll be speaking at FITC Toronto this Monday with Mathieu Badimon. We’ll be speaking on Firstborn’s behalf. Our presentation is titled “Tools & Prototypes: Crafting a Unique Digital Experience.” It’s basically focusing on how for each project we make, we usually need to make demos to test techniques or make tools to aid in development. It’s more of a creative presentation than a technical one, as we really won’t be talking about code directly. So if you’re up at FITC this weekend, we’ll be presenting Monday at 12:10.
I made some updates to my time displacement experiments. I lumped them all into one nice little app, as well as give some options to toggle the effect. I also added the option to choose video source, so it doesn’t have to be with a webcam anymore. I added the option to tint what area is being effected, as well as experimented with frame blending to try to lessen hard edges. Like the previous post, I’m still using Pixel Bender to apply the displacement.
I decided to try using Pixel Bender to do the time displacement in the recent experiments I’ve been doing, as I was hoping it would perform better. It actually seems to, but there still is a significant lag. However, I’m pretty sure this is because of having to make a huge sprite sheet first.
I basically make a huge sprite sheet of all the frames I have saved. Since Flash 10 lets you have bigger bitmaps (16,777,215 pixels total, max dimension at 8,191 src) I can create a large image that tiles all the frames I currently have saved. I can have a max of 12 frames across (at 640) and 4 frames down (at 480). So after I create/update this giant bitmap, I send it to a pixel bender filter that is currently applied to a gray scale image. The Pixel Bender filter then looks at each pixel, evaluates the brightness and interpolates it to what frame it needs to update from, then replaces the current pixel with the appropriate one from the selected frame.
These experiments are still a little rough, but I’ll post ’em anyways. No videos yet, I need to find some time to do screen recordings. The ticker is obviously by Mr. Doob. You’ll need a webcam for all of these, sorry.
- brightness: Displacement based off of brightness of current frame.
- movement: Displacement based off of movement (difference between frames)
- painting: Displacement based off area pained by user, as in previous post (inspired by Smudge City).
- noise: Displacement based off of perlin noise.
- filmstrip: Displacement based off of titled blocks.
In any of these experiments, you can right-click to show the source that is controlling the displacement.
I started playing around with doing some experiments with video and time delay. When showing the previously mentioned JPEG video experiment to my coworker Roushey, he mentioned how just the effect of the time delay was interesting. I was thinking about neat things to do with, and the first idea I had was to offset each frame as one row or column of pixels. So in a 640 x 480 video, the left-most column of pixels would be from the current frame, and the last rightmost column would be from 639 frames ago. The experiment is live here, and there’s a video of it on YouTube.
A few days ago I was thinking if there was a way to reproduce the effect of JPEG compression real time in flash. I figured the best way would be to literally encode the image with the native AS3 JPEGEncoder class. The problem there was that the returned ByteArray could not be interpreted as an image anymore. However, the Loader class does allow for loading a ByteArray and converting it to a Bitmap. And luckily, the loadBytes method let us load the ByteArray right from Flash (as opposed to having to save out a file and load it externally).