Modeling smooth and/or linear  Example Based Image Synthesis  Example Based Image Synthesis

Introduction

Image-based view synthesis is an important application of learning networks, offering the ability to render realistic images without requiring detailed models of object shape and illumination effects. To date, much attention has been given to the problem of view synthesis under varying camera pose or rigid object transformation. Several successful solutions have been proposed in the computer graphics and vision literature, including view morphing [12], plenoptic modeling/depth recovery [8], ``lightfields'' [7], and recent approaches using the trifocal tensor for view extrapolation [13].

For non-rigid view synthesis, networks for model-based interpolation and manifold learning have been used successfully in some cases [14,2,4,11]. Techniques based on Radial Basis Function (RBF) interpolation or on Principle Components Analysis (PCA), have been able to interpolate face images under varying pose, expression and identity [1,5,6]. However, these methods are limited in the types of object appearance they can accurately model. PCA-based face analysis typically assumes images of face shape and texture fall in a linear subspace; RBF approaches fare poorly when appearance is not a smooth function.

We want to extend non-rigid interpolation networks to handle cases where appearance is not a linear manifold and is not a smooth function, such as with articulated bodies. The mapping from parameter to appearance for articulated bodies is often one-to-many due to the multiple solutions possible for a given endpoint. It will also be discontinuous when constraints call for different solutions across a boundary in parameter space, such as the example shown in Figure 1.

Our approach represents an appearance mapping as a set of piecewise smooth functions. We search for sets of examples which are well approximated by the examples on the convex hull of the set's parameter values. Once we have these 'safe' sets of examples we perform interpolation using only the examples in a single set.

The clear advantage of this approach is that it will prevent inconsistent examples from being combined during interpolation. It also can reduce the number of examples needed to fully interpolate the function, as only those examples which are on the convex hull of one or more example sets are needed. If a new example is provided and it falls within and is well-approximated by the convex hull of an existing set, it can be safely ignored.

The remainder of this paper proceeds as follows. First, we will review methods for modeling appearance when it can be well approximated with a smooth and/or linear function. Next, we will present a technique for clustering examples to find maximal subsets which are well approximated in their interior. We will then detail how we select among the subsets during interpolation, and finally show results with both synthetic and real imagery.
 
 

 

Figure 1:   Synthesized examples of an appearance mapping (endpoint location to contour shape) for a 2DOF planar arm. Discontinuities in appearance due to workspace constraints make this a difficult function to learn from examples; the first and last example are very close in parameter space, but far in appearance space. 

 

        

Figure 2:   Arm appearance interpolated from examples using approximation network. (a) shows results using all examples in a single network; (b) using the example sets algorithm described in text. Note poor approximation on last two examples in (a); appearance discontinuities and extrapolation cause problems for full network, but are handled well in examples sets method. 

(a)   (b) 

 

    
9/11/1998