View-based image interpolation is a powerful paradigm for generating realistic imagery without full models of the underlying scene geometry. Current techniques for non-rigid interpolation assume appearance is a smooth function. We apply an example clustering approach using on-line cross validation to decompose a complex appearance mapping into sets of examples which can be smoothly interpolated. We show results on real imagery of human arms, with correspondences recovered from deformable contour tracking. Given images of an arm moving on a plane with various configuration conditions (elbow up and elbow down), and with associated parameter vectors marking the hand location, our method is able to discover a small set of manifolds with a small number of exemplars each can render new examples which are always physically correct. A single interpolating manifold for this same data has errors near the boundary between different arm configurations, and where multiple images have the same parameter value.