Probabilistic image-based rendering with Gaussian mixture model

One major challenge in traditional image-based rendering is 3D scene reconstruction by estimating accurate dense depth map, which suffers from the ambiguities in textureless or periodically textured regions. Alternatively, statistical methods may be used to estimate a most likely color for each pixel for photorealistic rendering from multiple views of the same scene. Such statistical methods normally require a relatively large number of input images to achieve reasonable quality for the synthesized image, if the estimation is purely nonparametric. In this paper, based on some reasonable assumptions on the configuration of the multiple views, we propose to use a two-component Gaussian mixture model for the appearance of a given pixel in all the views so that both the problem of occlusion and the problem of noise can be considered simultaneously. Then we use the Expectation-Maximization algorithm to estimate the model parameters. The virtual pixel is given as a maximum likelihood estimate for one of the mixture components. Experiments shows that reasonable performance can be obtained even with only a few input images.