Model-Based Matching by Linear  Combinations of Prototypes (Michael Jones and Tomaso Poggio)

• Overview:
• 2 Versions:
• One that handles Line Drawings
• One that handles Grayscale images
They are basically the same, only difference is the second one contains a texture term in its combination equation...(more later)
 
• Top Level:

 
• Having a set of prototypes (a set of images which show how an object can change), and a novel (unknown) image of the same class, define a BASE image.





• Using this Base Image, generate "flow fields" between it and all other prototype images (only done once per database)
• Create "Image Pyramid" for the novel Image and base image.
• For each level in the pyramid:
• estimate the linear coefficients and the affine parameters
• go to next level
• output coefficients and parameters

 
• In Depth:

• Model Generation:

Storing a set of N+1 images {I0,..,IN}, from same class, which show how image can change (facial expressions, car outlines), called  prototypes.  Decide on a base image, i.e. an image which is generally considered to be the least altered (hand chosen).  This base image is called the reference prototype ( generally I0 ).

The pixelwise correspondence is generated between EVERY prototype and the reference prototype.  This is done by determining vector needed to move every pixel from reference prototype (X, Y) to prototype being considered.

Where (Xnew, Ynew) is the point in the current prototype which corresponds to (X, Y) in the reference prototype.  Sj  outputs a correspondence field for every image in database.

For each image, this correspondence Field is represented by two matrices, Dx and Dy.    These represent the displacement for each image to the reference prototype in the X, and Y direction, respectively.

Compute Dx' and Dy' which are the sums of all the prototype's Dx and Dy,   Multiplied by a linear coefficient (Ci), and  an affine parameter (Pi) to handle small transformations (translations, scales, rotations).

To get and image I', using these flow fields ( Dx' and Dy' ), the reference prototype is warped along these matrices...

• Texture Handling for Images

To handle grayscale images, a simple addition is made.  A "texture vector" is computed (not really considering texture though, just pixel intensity).  It is used in a similar manner to the shape vector (flow fields above), i.e. maps the texture of the reference image to the location in the current prototype... (not too concerned with it, more  focused on shape based recognition).
 
• Fitting Novel (Unknown) Images


Once the flow fields are generated over every image, the correspondence between the prototypes and the unknown image needs to be found.  To do this, an Error between the novel image and the current guess for closest prototype is defined.  The process is to minimize the error with respect to the linear coeff's (Ci) and the affine parameters (Pi) .

Use sum of square differences error:

Basically, just the difference between the novel image and the warped base image.

To minimize the error related to Ci and Pi , use Levenberg-Marquardt algorithm, which takes the derivatives of the error functional with respect to each parameter (i.e. Ci and Pi ), this outputs the optimal Ci and Pi .