MatCLIP: Light- and Shape-Insensitive Assignment of PBR Material Models

Michael Birsak1, John Femiani2, Biao Zhang1, Peter Wonka1
1King Abdullah University of Science and Technology, KSA
2Miami University, USA
SIGGRAPH 2025
Paper (arXiv) Code (coming soon)
Grapefruit slice atop a pile of other slices

(Left) Latent Diffusion Models (LDMs) are a powerful tool to obtain good material assignments, but the output is RGB only. (Middle) Our MatCLIP model robustly matches a PBR material from a material database to each part of the shape. (Right) By comparison, procedurally assigning PBR materials from meaningful substance categories to 3D shapes yields unconvincing results.

Abstract

Assigning realistic materials to 3D models remains a significant challenge in computer graphics. We propose MatCLIP, a novel method that extracts shape- and lighting-insensitive descriptors of Physically Based Rendering (PBR) materials to assign plausible textures to 3D objects based on images, such as the output of Latent Diffusion Models (LDMs) or photographs. Matching PBR materials to static images is challenging because the PBR representation captures the dynamic appearance of materials under varying viewing angles, shapes, and lighting conditions. By extending an AlphaCLIP-based model on material renderings across diverse shapes and lighting, and encoding multiple viewing conditions for PBR materials, our approach generates descriptors that bridge the domains of PBR representations with photographs or renderings, including LDM outputs. This enables consistent material assignments without requiring explicit knowledge of material relationships between different parts of an object. MatCLIP achieves a top-1 classification accuracy of 76.6\%, outperforming state-of-the-art methods such as PhotoShape and MatAtlas by over 15 percentage points on publicly available datasets. Our method can be used to construct material assignments for 3D shape datasets such as ShapeNet, 3DCoMPaT++, and Objaverse.

Overview

Grapefruit slice atop a pile of other slices

(left) Our method learns robust material descriptors through diverse renderings across geometric shapes and lighting conditions. (bottom center) MatCLIP aligns material embeddings with corresponding parts of 3D objects using cosine similarity and attention mechanisms. (right) We train our MatCLIP model using a large dataset of masked renderings. (top center) We leverage the output from LDMs (e.g., Stable Diffusion) to obtain meaningful material assignments to untextured shape collections.

Result Videos

We conduct qualitative evaluations using outputs from FLUX to assess MatCLIP's ability to generalize beyond the closed set of training materials. These evaluations focus on assigning materials to synthetic target images generated using depth, edge, and text conditioning, as described. Our model selects the best-matching material from the MatSynth database for each target image, producing visually coherent and contextually appropriate assignments, even for complex shapes and lighting conditions.

Result Images (FLUX)

For each example, the left image is the output image from FLUX, the right image shows a rendering of the corresponding shape with materials assigned by our MatCLIP model.

Result Images (Stable Diffusion)

For each example, the left image is the output image from Stable Diffusion, the right image shows a rendering of the corresponding shape with materials assigned by our MatCLIP model.

Limitations

Currently, our method assigns materials based on only a single input image. While extending our method to multiview would be straightforward, the even safer and easier option to obtain a coherent result is to propagate materials from visible to invisible parts based on adjacency information and part semantics. Here, we just want to highlight the problem by assigning a random non-fitting material to shape parts which are invisible in the input image.

MY ALT TEXT

BibTeX


        @article{birsak2025matclip,
          author    = {Birsak, Michael and Femiani, John and Zhang, Biao and Wonka, Peter},
          title     = {MatCLIP: Light- and Shape-Insensitive Assignment of PBR Material Models},
          journal   = {arXiv preprint arXiv:2501.15981},
          year      = {2025},
        }