Portrait Virus Mutations

StackFilter1 Movie

2010-10-03T15:12:00.001-07:00

StackFilter 1 from john dalton on Vimeo.

I've recently been trying some experiments using stack filtering to create movies. This particular example stackfilters the spatially aligned siggraph 2010 portrait virus paintings with a series of different temporal image processing operations in Studio Artist 4.

Here's a link to the full size video on my vimeo site. If you search on the stackfilter tag there you can check out other examples of constructing animations based on stack filtering effects.

Mutation 51

2010-10-01T06:13:00.000-07:00

I've got 2 mutations here, the first based on the spatially normalized self portraits and the second on the original self portrait stack. A temporal slit scan through the entire portrait stack was used for both examples. Note how the normalization makes a big difference to get feature coherence in the final output for this kind of processing.

Mutation 50

2010-09-30T06:10:00.000-07:00

A seven frame window through the original self portrait stack was used as input to a temporal rank min-max stack filter to generate this portrait.

Mutation 49

2010-09-29T06:09:00.000-07:00

A seven frame temporal processing window was used with the temporal rank median filter to generate this portrait from the original self portrait stack.

Mutation 47

2010-09-28T06:05:00.000-07:00

This portrait was generated by stack filtering the original set of self portraits with the temporal difference1 filter.

Mutation 46

2010-09-27T06:03:00.000-07:00

This portrait was generated by stack filtering the original set of self portrait images with the temporal difference1 filter.

Mutation 45

2010-09-26T06:58:00.000-07:00

We're going to switch gears for awhile and use the original self portrait stack as opposed to the spatially normalized self portrait stack. This portrait was generated with a paint action sequence that auto draws the eyes and mouth of every portrait in the input stack.

So it's an example of using a repetitive process encoded in a PASeq. All of the processing is done automatically, including the face feature detection. The diversity of the original portrait set is desired for this kind of processing (as opposed to the set of previous examples where it was desirable for the painted facial features to be spatially normalized.

Mutation 44

2010-09-25T06:56:00.000-07:00

This portrait was generated by stack filtering the spatially normalized set of self portraits with the temporal difference1 filter.

Mutation 43

2010-09-24T06:54:00.000-07:00

This portrait was generated by processing the spatially normalized self portraits with a temporal rank neighborhood filter.

Mutation 42

2010-09-23T06:52:00.000-07:00

This portrait was generated by stack filtering the spatially aligned self portraits with a different temporal rank neighborhood filter preset.

Mutation 41

2010-09-22T06:50:00.000-07:00

This portrait was generated by stack filtering the spatially normalized self portraits with the temporal rank neighborhood filter.

Mutation 40

2010-09-21T06:48:00.000-07:00

This portrait was generated by stack filtering the spatially normalized slef portraits with the temporal scan tracker 2 smart seam processing using a horizontal scan and a wide temporal processing window.

Mutation 39

2010-09-20T06:46:00.000-07:00

This portrait was generated by stack filtering the spatially normalized self portraits with the temporal scan tracker 2 smart seam processing, using a vertical scan direction.

Mutation 38

2010-09-19T06:44:00.000-07:00

This portrait was generated by stack filtering the spatially normalized self portrait stack with the temporal scan tracker 2 processing, using the new smart seam feature.

Mutation 37

2010-09-18T06:41:00.000-07:00

This portrait was generated by stack filtering the spatially normalized self portraits with the temporal difference1 filter.

Mutation 38

2010-09-17T06:38:00.000-07:00

This image was generated by processing the spatially aligned portraits with a temporal displacement filter. The displacement is based on one of the normalized self portraits chosen at random.

Mutation 37

2010-09-16T06:36:00.000-07:00

This portrait was generated by processing the spatially normalized self portraits with the temporal motion rank min filter.

Mutation 36

2010-09-15T06:34:00.000-07:00

This portrait was generated by stack filtering the spatially normalized set of self portraits with the temporal motion rank max filter.

Mutation 35

2010-09-14T06:31:00.001-07:00

This portrait was generated by processing the complete set of self-aligned spatially normalized portraits with the temporal rank mean difference mask filter.

Mutation 34

2010-09-13T06:30:00.000-07:00

This portrait image was generated by processing the complete set of spatially aligned self-normalized portraits with the temporal rank MinMask filter.

Mutation 33

2010-09-12T06:27:00.000-07:00

This image was generated by stack filtering the complete set of spatially normalized self portraits with the temporal rank median maximum distance filter.

Mutation 32

2010-09-11T06:12:00.000-07:00

The daily portrait virus mutation posts are going to switch gears slightly for awhile. This new set is based using Michael's daily self portrait virus blog imagery as the source for stack filtering. As opposed to the Siggraph 2010 portrait virus imagery that has been used to date.

The other factor that is slightly different is that during the process of running this whole portrait virus mutation experiment we found a bug in the inverse warp code. I wasn't quite satisfied with the spatially aligned imagery being generated by the stack filtering algorithms. And indeed once the inverse warp code was improved the results of spatially aligned stack filtering did as well.

For example, the image above was generated by processing all of Michael's self portrait paintings to date with a temporal mean filter. So you can think of the image as being the generic self portrait. It's like the average portrait, but with all of the face paintings mutually aligned to a standardized portrait (one of the portrait virus images selected at random). The temporal median is also very similar (it isn't always the case).

There's a Studio Artist news blog post on stack filtering portrait images that explains the principals behind how this image was generated (and stack filtering in general) in detail.

At some point i may return to the Siggraph 2010 portrait virus images and redo some of the spatially aligned results shown here, because i think they would be more in line with how i originally thought they would turn out. I had thought about redoing all of the old posts, but discussed this in a Studio Artist Forum post. Part of the point of generating artistic imagery is that mistakes may be just as interesting as technically correct results (sometimes more so). So leaving things the way they progressed as this project developed seemed to make more sense.

Mutation 31

2010-09-10T06:12:00.000-07:00

Another example of applying the temporal difference filter to the spatially aligned portrait virus image stack with a limited temporal window.

Mutation 30

2010-09-09T06:08:00.000-07:00

Temporal rank neighborhood with the max setting. So it's looking for maximum energy in spatially local neighborhoods across the entire stack (the spatially aligned portrait stack in this case).

Mutation 29

2010-09-08T06:04:00.000-07:00

Another smart seam effect using the temporal scan track. I'm using a vertical scan this time as opposed to a horizontal scan. With a limited temporal range, they get a little too venetian blind like for my tastes when you use the entire stack.

Mutation 28

2010-09-07T06:02:00.000-07:00

Trying some experiments with the temporal scan track. This is using the new smart seam technology for building the transitions. Again with the spatially aligned portrait virus image stack as input.

Mutation 28

2010-09-06T18:58:00.000-07:00

This was generated by processing the spatially aligned portrait virus image stack with the temporal rank variance filter. Again, i'm using temporal processing windows that are smaller than the entire sequence to get more interesting results (to my eye).

Mutation 27

2010-09-05T06:53:00.000-07:00

Another example of working with the temporal difference filter. By modifying the temporal processing window and the effect parameters you can generate a lot of different interesting imagery. We're again using the spatially aligned portrait virus image stack because we're interested in building composite facial images based on the variations in the paint strokes associated with the different portraits. If we used the original portrait stack that isn't spatially aligned the temporal processing would pick up on the differences associated with where the objects in the various portraits are as opposed to focusing on the differences in how the faces themselves are painted.

Mutation 26

2010-09-04T06:51:00.001-07:00

An example of processing the spatially aligned portrait virus image stack with the temporal difference filter.

Mutation 25

2010-09-03T06:48:00.000-07:00

Another example of working with the temporal difference matte effect. This time processing the spatially aligned portrait virus image stack.

Mutation 24

2010-09-02T06:46:00.000-07:00

An example of using a temporal median maximum distance filter on the spatially aligned portrait virus image stack.

Mutation 23

2010-09-01T06:42:00.000-07:00

I'm using a temporal min-max filter for this particular example. Again, i'm using a limited temporal processing window with the spatially aligned portrait virus image stack.

Mutation 22

2010-08-31T06:39:00.000-07:00

An example of using temporal rank median filtering on the spatially aligned stack of portrait virus images. I'm using limited temporal windows for processing for all of these recent posts (as opposed to processing the entire stack of images).

Mutation 21

2010-08-30T06:38:00.000-07:00

Another example of using temporal rank max filter on the spatially aligned stack of portrait virus images.

Mutation 20

2010-08-29T06:34:00.000-07:00

One of the advantages of working with the spatially normalized set of portrait images is that you can start to pull out features associated with the portraits in a more intelligent way since all of the facial images are spatially aligned across the stack. Here's an example using temporal rank max filtering on the spatially normalized portrait stack..

Mutation 19

2010-08-28T06:26:00.000-07:00

The temporal pixel jumble effect applied to the complete stack of original portraits. Not particularly effective in my opinion, but i thought i'd post one example to show off what it does. I guess there is some rough similarity to the temporal mean rank filter effect in a more pixelated way.

Mutation 48

2010-08-28T06:07:00.000-07:00

This portrait was generated by processing the original set of self portraits with the temporal motion rank max filter.

Mutation 18

2010-08-27T06:43:00.000-07:00

There's a ton of stuff that can be done by treating the stack of source imagery as a movie brush for building photo mosaics. We'll get more heavily into that later but i thought i'd post a really simple example. I modified the 3.5 Collection Graffiti category to use the original portrait virus images. I used a different brush load image processing algorithm for the colorization, the meanshift1 algorithm seems to have some issues with the dark areas of the siggraph 2010 portraits. So i'll look into that, not sure if it's a bug or an artifact associated with the compression i used for the movie brush.

I think movie background textures may end up being more effective for this set of source imagery since you can use the source portraits at full resolution in the generated texturing to get at all of the detail of the original paint stroke texturing. I had to size the portraits down so small to make a source movie brush from the original portrait stack that you lose most of the paint effects in the individual portraits that make up the mosiac.

Mutation 17

2010-08-26T06:41:00.000-07:00

Temporal motion rank filter with the min setting. So we're looking at minimum motion in local neighborhoods across the sequence of original portraits.

Mutation 16

2010-08-25T06:15:00.000-07:00

A different set of temporal difference matte parameters applied to the original stack of portrait images. Looking for similarity as opposed to variation in the sequence of portrait imagery.

Mutation 15

2010-08-24T06:12:00.000-07:00

The temporal difference matte is a pretty versatile effect, depending on which settings you choose. I'm using the original stack of portraits as input for this effect.

Mutation 14

2010-08-23T06:05:00.000-07:00

A good example of a failed experiment. I loved the earlier results i was getting painting of of source edge maps, so i thought i'd use the Color Edge ip op to min composite source edge maps across the entire portrait stack and then interpolate in the white spaces. A big mess in my opinion. Compare it to the original approach below where i paint the generated paths while masking out areas already painted. The masking is the key to avoiding too much build up.

Mutation 13

2010-08-22T06:01:00.000-07:00

A real simple variation on building PASeq that processes the entire stack of portraits. I'm using the original portrait stack and just painting with partially transparent circles. I ran the PASeq twice on the stack to build up the image density.

Mutation 12

2010-08-21T06:55:00.000-07:00

Another variation on the theme of building a PASeq that processes the entire stack of portraits to generate a final output image. This one works by building source edge maps and then sketching them in. Surprisingly different than some of the other stack filter effects.

Mutation 11

2010-08-20T06:51:00.000-07:00

Variation on the previous eye detection experiment, but this time i'm using automatic face detection and then matting the results across the stack with some partial transparency. So i use a PASeq to process the entire stack of portraits to generate the final single image. I used the original portrait stack so that the variation in face positioning would help build up the composite face image.

Mutation 10

2010-08-19T06:46:00.000-07:00

Another temporal scan tracker image using a limited range of the overall portrait stack. I'm using the spatially normalized stack in this case. I'm also using a diagonal scan.

Mutation 9

2010-08-18T06:23:00.000-07:00

Another example generated with the temporal scan tracker. This time i used the spatially aligned portraits and a smaller number of frames. I think that works better for this set of stack images, it gets rid of the venetian blind effect and generates some interesting composite heads. Depending on where you start the scan in the image sequence you can generate a variety of different effects. Again, i'm blending with maximum energy as opposed to a linear blend.

Mutation 8

2010-08-17T06:14:00.000-07:00

Slit scan effect generated by using the temporal scan tracker on the entire stack of original portraits. I'm using maximum energy blending. You get a tighter head when you use the spatially aligned portrait stack, i'll post some of those later.

Mutation 7

2010-08-16T06:00:00.000-07:00

This image was generated by using the temporal displacement operation. I started with a set of randomly colored rectangles in a grid. The grayscale values in the squares map a displacement into the stack of portrait frame images. There's an infinite number of different effects like this you can generate. This image was generated off of the original portrait stack.

Mutation 6

2010-08-15T06:56:00.000-07:00

This image uses another temporal motion rank filter, but it's choosing the pixel with the minimum motion for each spatial location in the stack of portraits. So it's related to yesterday's post, but kind of the inverse effect. It accents similarity as opposed to difference in adjacent images in the portrait stack.

Mutation 5

2010-08-14T06:42:00.000-07:00

This image was generated by using the temporal motion rank filter with the spatially normalized stack or portrait virus paintings. So it analyzes the stack of images and displays the pixel with the maximum motion at each spatial location in the output canvas. Obviously the ordering of the images in the stack is going to influence the result of this kind of processing. So by shuffling the ordering of the images in the stack sequence you could generate different variations of the effect.

Mutation 4

2010-08-13T06:53:00.000-07:00

This image is an example that was built from a spatially normalized stack of portrait virus images. By spatial normalization i mean that every painted portrait was spatially warped to a standard portrait facial position. The standard position was just one portrait that i randomly picked out of the 2010 Siggraph portrait virus set. Every other portrait was spatially warped to match the standard position. I'll discuss the mechanics of that process at a later time. The resulting image was generated with the temporal rank filter and can be thought of as the average of all of the portrait virus images. So you could think of it as the generic portrait derived from the entire set.

I've generated another image below that uses the same process, but with the original non-spatially normalized portraits as the input stack. Note that it's much more of a blur and doesn't capture as much of the facial distinctions and detailing of the overall set of portraits.

It does tell us something about the overall set of original portraits (from the standpoint of visualization analysis), mainly that in average the faces do tend to be centered. And for some temporal processing effects that spatial variation of the individual portraits can lead to visually interesting results. But for other processing effects working with the spatially normalized image set can lead to very different results.

Mutation 3

2010-08-12T06:43:00.000-07:00

Thought i'd try an experiment with eye detection. The stack of portraits are processed with a PASeq that auto-selects the painted eyes and then paints those into the canvas. So the eyes build up over time.

So this one is a little different than the stack filter effects that use temporal image processing. Building a PASeq that processes a stack of images to build up a single canvas can be thought of as a different approach to stack filtering. We'll get back to time-based processing tomorrow, but it's important to realize that there's many different approaches you can take to build stack filtering effects.

Again, i'm using the original series of 2010 Siggraph portrait virus images for this example. We will be discussing building a source image series spatially warped to a standard image. If we had used that sequence for this particular effect all of the eyes would be in the same position, so it would not be as effective. However, for other kinds of time based processing the standardized sequence can be very useful. We will show an example of that tomorrow.

Mutation 2

2010-08-11T06:09:00.000-07:00

Generated with temporal rank neighborhood filtering in Studio Artist 4.

Again, we're working with the original stack of portrait virus images as the input for this effect.

Mutation 1

2010-08-10T06:01:00.000-07:00

Generated by processing a stack of portrait virus images with a StudioArtist temporal image processing effect.

This first series is based on the original portrait virus images generated at Siggraph 2010. Later we'll be getting into modified image stacks based on spatially warping the stack of images to a 'standard' image.

What is Portrait Virus Mutations?

2010-08-09T21:25:00.000-07:00

Michael Wright has been running his Portrait Virus project for several years now at various Siggraph shows. All of the portrait virus images are created using Studio Artist. The snapshot above shows Michael standing in front of a collection of portrait virus images generated during a previous Siggraph show in the old Guerilla Studio. (There's also a portrait of me buried in there if you look closely). Examples of the Portrait Virus from Siggraph Studio 2010 are available online here.

Michal runs Studio Artist on a portable computer in the Siggraph studio. He uses Studio Artist's video source capture option to grab a video still of someone in the gallery, and then uses Studio Artist's paint synthesizer in real time with a Wacom tablet to generate a painted digital portrait of that person. Michael also has a daily portrait virus post page available here that shows a new Studio Artist generated self-portrait every day. I'm a big believer in this kind of 'daily art post' project because i think the discipline of generating a new art image and posting it every day helps stimulate creativity in many different ways.

I thought it would be fun to try generating my own portrait virus mutation project. All of the source imagery in this initial portrait virus mutation project is generated from collections of Michael's portrait virus imagery. Part of the point of the mutation project is that all of the mutated images are derived from multiple input images. As opposed to processing a single input image. There are a number of different approaches one can use to work with collections of images with the goal of generating single output images from multiple input images. Photo mosaic imagery is one example of this kind of approach. Another approach is what i call 'stack filtering'. 'Stack' refers to a stack of images.

Studio Artist 4 has a number of different temporal image processing operations. Temporal image operations are image processing effects that work off of multiple frame input images at different times to generate a single output image. They are really designed for video processing. But there's no reason why you can't take any stack of images and run them through a temporal processing effect. So stack filtering is referring to taking a stack of images and processing them with a temporal video processing effect.

I'll be posting different portrait virus mutations on this blog, one a day, until the project runs it's course. The initial posts are going to focus on the use of Studio Artist temporal image operations to process stacks of Michael's original portrait virus images, so again a collection of multiple images are processed to generate a single processed or mutated output image. Hopefully this project will give you some ideas for using stack filtering in your own artistic endeavors. The creative possibilities are really endless.

A big thank you to Michael for being kind enough to allow me to mutate his original portrait virus imagery and infect it with a new digital virus. Michael is a long time pioneer in digital art, and you can learn more about his body of work here.