StackFilter1 Movie

StackFilter 1 from john dalton on Vimeo.

I've recently been trying some experiments using stack filtering to create movies. This particular example stackfilters the spatially aligned siggraph 2010 portrait virus paintings with a series of different temporal image processing operations in Studio Artist 4.

Here's a link to the full size video on my vimeo site. If you search on the stackfilter tag there you can check out other examples of constructing animations based on stack filtering effects.

Mutation 51

I've got 2 mutations here, the first based on the spatially normalized self portraits and the second on the original self portrait stack. A temporal slit scan through the entire portrait stack was used for both examples. Note how the normalization makes a big difference to get feature coherence in the final output for this kind of processing.

Mutation 50

A seven frame window through the original self portrait stack was used as input to a temporal rank min-max stack filter to generate this portrait.

Mutation 49

A seven frame temporal processing window was used with the temporal rank median filter to generate this portrait from the original self portrait stack.

Mutation 47

This portrait was generated by stack filtering the original set of self portraits with the temporal difference1 filter.

Mutation 46

This portrait was generated by stack filtering the original set of self portrait images with the temporal difference1 filter.

Mutation 45

We're going to switch gears for awhile and use the original self portrait stack as opposed to the spatially normalized self portrait stack. This portrait was generated with a paint action sequence that auto draws the eyes and mouth of every portrait in the input stack.

So it's an example of using a repetitive process encoded in a PASeq. All of the processing is done automatically, including the face feature detection. The diversity of the original portrait set is desired for this kind of processing (as opposed to the set of previous examples where it was desirable for the painted facial features to be spatially normalized.

Mutation 44

This portrait was generated by stack filtering the spatially normalized set of self portraits with the temporal difference1 filter.

Mutation 43

This portrait was generated by processing the spatially normalized self portraits with a temporal rank neighborhood filter.

Mutation 42

This portrait was generated by stack filtering the spatially aligned self portraits with a different temporal rank neighborhood filter preset.

Mutation 41

This portrait was generated by stack filtering the spatially normalized self portraits with the temporal rank neighborhood filter.

Mutation 40

This portrait was generated by stack filtering the spatially normalized slef portraits with the temporal scan tracker 2 smart seam processing using a horizontal scan and a wide temporal processing window.

Mutation 39

This portrait was generated by stack filtering the spatially normalized self portraits with the temporal scan tracker 2 smart seam processing, using a vertical scan direction.

Mutation 38

This portrait was generated by stack filtering the spatially normalized self portrait stack with the temporal scan tracker 2 processing, using the new smart seam feature.

Mutation 37

This portrait was generated by stack filtering the spatially normalized self portraits with the temporal difference1 filter.

Mutation 38

This image was generated by processing the spatially aligned portraits with a temporal displacement filter. The displacement is based on one of the normalized self portraits chosen at random.

Mutation 37

This portrait was generated by processing the spatially normalized self portraits with the temporal motion rank min filter.

Mutation 36

This portrait was generated by stack filtering the spatially normalized set of self portraits with the temporal motion rank max filter.

Mutation 35

This portrait was generated by processing the complete set of self-aligned spatially normalized portraits with the temporal rank mean difference mask filter.

Mutation 34

This portrait image was generated by processing the complete set of spatially aligned self-normalized portraits with the temporal rank MinMask filter.

Mutation 33

This image was generated by stack filtering the complete set of spatially normalized self portraits with the temporal rank median maximum distance filter.

Mutation 32

The daily portrait virus mutation posts are going to switch gears slightly for awhile. This new set is based using Michael's daily self portrait virus blog imagery as the source for stack filtering. As opposed to the Siggraph 2010 portrait virus imagery that has been used to date.

The other factor that is slightly different is that during the process of running this whole portrait virus mutation experiment we found a bug in the inverse warp code. I wasn't quite satisfied with the spatially aligned imagery being generated by the stack filtering algorithms. And indeed once the inverse warp code was improved the results of spatially aligned stack filtering did as well.

For example, the image above was generated by processing all of Michael's self portrait paintings to date with a temporal mean filter. So you can think of the image as being the generic self portrait. It's like the average portrait, but with all of the face paintings mutually aligned to a standardized portrait (one of the portrait virus images selected at random). The temporal median is also very similar (it isn't always the case).

There's a Studio Artist news blog post on stack filtering portrait images that explains the principals behind how this image was generated (and stack filtering in general) in detail.

At some point i may return to the Siggraph 2010 portrait virus images and redo some of the spatially aligned results shown here, because i think they would be more in line with how i originally thought they would turn out. I had thought about redoing all of the old posts, but discussed this in a Studio Artist Forum post. Part of the point of generating artistic imagery is that mistakes may be just as interesting as technically correct results (sometimes more so). So leaving things the way they progressed as this project developed seemed to make more sense.

Mutation 31

Another example of applying the temporal difference filter to the spatially aligned portrait virus image stack with a limited temporal window.

Mutation 30

Temporal rank neighborhood with the max setting. So it's looking for maximum energy in spatially local neighborhoods across the entire stack (the spatially aligned portrait stack in this case).

Mutation 29

Another smart seam effect using the temporal scan track. I'm using a vertical scan this time as opposed to a horizontal scan. With a limited temporal range, they get a little too venetian blind like for my tastes when you use the entire stack.

Mutation 28

Trying some experiments with the temporal scan track. This is using the new smart seam technology for building the transitions. Again with the spatially aligned portrait virus image stack as input.

Mutation 28

This was generated by processing the spatially aligned portrait virus image stack with the temporal rank variance filter. Again, i'm using temporal processing windows that are smaller than the entire sequence to get more interesting results (to my eye).

Mutation 27

Another example of working with the temporal difference filter. By modifying the temporal processing window and the effect parameters you can generate a lot of different interesting imagery. We're again using the spatially aligned portrait virus image stack because we're interested in building composite facial images based on the variations in the paint strokes associated with the different portraits. If we used the original portrait stack that isn't spatially aligned the temporal processing would pick up on the differences associated with where the objects in the various portraits are as opposed to focusing on the differences in how the faces themselves are painted.

Mutation 26

An example of processing the spatially aligned portrait virus image stack with the temporal difference filter.

Mutation 25

Another example of working with the temporal difference matte effect. This time processing the spatially aligned portrait virus image stack.

Mutation 24

An example of using a temporal median maximum distance filter on the spatially aligned portrait virus image stack.

Mutation 23

I'm using a temporal min-max filter for this particular example. Again, i'm using a limited temporal processing window with the spatially aligned portrait virus image stack.

Mutation 22

An example of using temporal rank median filtering on the spatially aligned stack of portrait virus images. I'm using limited temporal windows for processing for all of these recent posts (as opposed to processing the entire stack of images).

Mutation 21

Another example of using temporal rank max filter on the spatially aligned stack of portrait virus images.

Mutation 20

One of the advantages of working with the spatially normalized set of portrait images is that you can start to pull out features associated with the portraits in a more intelligent way since all of the facial images are spatially aligned across the stack. Here's an example using temporal rank max filtering on the spatially normalized portrait stack..

Mutation 19

The temporal pixel jumble effect applied to the complete stack of original portraits. Not particularly effective in my opinion, but i thought i'd post one example to show off what it does. I guess there is some rough similarity to the temporal mean rank filter effect in a more pixelated way.

Mutation 48

This portrait was generated by processing the original set of self portraits with the temporal motion rank max filter.

Mutation 18

There's a ton of stuff that can be done by treating the stack of source imagery as a movie brush for building photo mosaics. We'll get more heavily into that later but i thought i'd post a really simple example. I modified the 3.5 Collection Graffiti category to use the original portrait virus images. I used a different brush load image processing algorithm for the colorization, the meanshift1 algorithm seems to have some issues with the dark areas of the siggraph 2010 portraits. So i'll look into that, not sure if it's a bug or an artifact associated with the compression i used for the movie brush.

I think movie background textures may end up being more effective for this set of source imagery since you can use the source portraits at full resolution in the generated texturing to get at all of the detail of the original paint stroke texturing. I had to size the portraits down so small to make a source movie brush from the original portrait stack that you lose most of the paint effects in the individual portraits that make up the mosiac.

Mutation 17

Temporal motion rank filter with the min setting. So we're looking at minimum motion in local neighborhoods across the sequence of original portraits.

Mutation 16

A different set of temporal difference matte parameters applied to the original stack of portrait images. Looking for similarity as opposed to variation in the sequence of portrait imagery.

Mutation 15

The temporal difference matte is a pretty versatile effect, depending on which settings you choose. I'm using the original stack of portraits as input for this effect.

Mutation 14

A good example of a failed experiment. I loved the earlier results i was getting painting of of source edge maps, so i thought i'd use the Color Edge ip op to min composite source edge maps across the entire portrait stack and then interpolate in the white spaces. A big mess in my opinion. Compare it to the original approach below where i paint the generated paths while masking out areas already painted. The masking is the key to avoiding too much build up.

Mutation 13

A real simple variation on building PASeq that processes the entire stack of portraits. I'm using the original portrait stack and just painting with partially transparent circles. I ran the PASeq twice on the stack to build up the image density.

Mutation 12

Another variation on the theme of building a PASeq that processes the entire stack of portraits to generate a final output image. This one works by building source edge maps and then sketching them in. Surprisingly different than some of the other stack filter effects.

Mutation 11

Variation on the previous eye detection experiment, but this time i'm using automatic face detection and then matting the results across the stack with some partial transparency. So i use a PASeq to process the entire stack of portraits to generate the final single image. I used the original portrait stack so that the variation in face positioning would help build up the composite face image.

Mutation 10

Another temporal scan tracker image using a limited range of the overall portrait stack. I'm using the spatially normalized stack in this case. I'm also using a diagonal scan.

Mutation 9

Another example generated with the temporal scan tracker. This time i used the spatially aligned portraits and a smaller number of frames. I think that works better for this set of stack images, it gets rid of the venetian blind effect and generates some interesting composite heads. Depending on where you start the scan in the image sequence you can generate a variety of different effects. Again, i'm blending with maximum energy as opposed to a linear blend.

Mutation 8

Slit scan effect generated by using the temporal scan tracker on the entire stack of original portraits. I'm using maximum energy blending. You get a tighter head when you use the spatially aligned portrait stack, i'll post some of those later.

Mutation 7

This image was generated by using the temporal displacement operation. I started with a set of randomly colored rectangles in a grid. The grayscale values in the squares map a displacement into the stack of portrait frame images. There's an infinite number of different effects like this you can generate. This image was generated off of the original portrait stack.