Peter Kocsis

NeuMatEx extracts neural materials from multi-view images using a hybrid approach: a Large Material Reconstruction Model provides a learned material prior and initialization, while differentiable inverse rendering refines the result across views. It reconstructs material effects beyond standard PBR, including haze, dust, clearcoat, fuzz, scattering, and mixtures of these effects, while keeping the resulting neural materials suitable for real-time path tracing.

We reconstruct high-quality physically based materials from multi-view images. Material reconstruction is highly underconstrained and typically relies on analysis-by-synthesis, which requires expensive and noisy path tracing. To better constrain the optimization, we incorporate single-view priors into the reconstruction process and propose a method to robustly aggregate a parametric function of the monocular predictions. Our results outperform state-of-the-art methods in material disentanglement on both synthetic and real scenes, producing sharp and clean reconstructions suitable for high-quality relighting.

We introduce direct PBR generation from text. Instead of training control modules to image generators, direct PBR generation allows for explicit control. First, we train separate LoRA modules for each intrinsic components. Then, we introduce cross-intrinsic attention with a rerendering loss to achieve coherent PBR generation. Our method outperforms text->image->PBR baselines. Our predictions can also be distilled into 3D scenes opening up large-scale PBR texture generation.

Recent generative methods lack lighting control, which is crucial to numerous artistic aspects of image generation such as setting the overall mood or cinematic appearance. To overcome these limitations, we propose to condition the generation on shading and normal maps. We model the lighting with single bounce shading, which includes cast shadows. We first train a shading estimation module to generate a dataset of real-world images and shading pairs. Then, we train a control network using the estimated shading and normals as input. Our method demonstrates high-quality image generation and lighting control in numerous scenes.

Intrinsic image decomposition is a highly ambigous task. Deep-learning-based methods often fail due to the lack of large-scale real world data. We propose to formulate the problem probabilistically and generate possible decompositions using a generative model. This way, we can also utilize the strong image prior of diffusion models for the task of material estimation, which largely helps generalization.

Convolutional neural networks were the standard for solving many computer vision tasks until recently, when Transformers of MLP-based architectures have started to show competitive performance. These architectures typically have a vast number of weights and need to be trained on massive datasets; hence, they are not suitable for their use in low-data regimes. In this work, we propose a simple yet effective framework to improve generalization from small amounts of data. We augment modern CNNs with fully-connected (FC) layers and show the massive impact this architectural change has in low-data regimes.