world

RenderFormer: Neural Rendering of Triangle Meshes with Global Illumination

admin

Jun 1, 2025 - 04:45

0 0

RenderFormer

Transformer-based Neural Rendering of Triangle Meshes with Global Illumination

SIGGRAPH 2025

¹State Key Lab of CAD and CG, Zhejiang University

²Microsoft Research Asia

³College of William & Mary

Introduction

We present RenderFormer, a neural rendering pipeline that directly renders an image from a triangle-based representation of a scene with full global illumination effects and that does not require per-scene training or fine-tuning.

Mesh to Image, End to End

Instead of taking a physics-centric approach to rendering, we formulate rendering as a sequence-to-sequence transformation where a sequence of tokens representing triangles with reflectance properties is converted to a sequence of output tokens representing small patches of pixels.

Simple Transformer Architecture with Minimal Prior Constraints

RenderFormer follows a two stage pipeline: a view-independent stage that models triangle-to-triangle light transport, and a view-dependent stage that transforms a token representing a bundle of rays to the corresponding pixel values guided by the triangle-sequence from the the view-independent stage. Both stages are based on the transformer architecture and are learned with minimal prior constraints. No rasterization, no ray tracing.

Rendering Gallery

Examples of scenes rendered with RenderFormer demonstrating various lighting conditions, materials, and geometric complexity, without any per-scene training or fine-tuning. Check out the reference images for more details.

Cornell Box

Source: Cornell University Program of Computer Graphics

Stanford Bunny in Cornell Box

Source: Stanford University Computer Graphics Laboratory

Lucy Statue

Source: Stanford University Computer Graphics Laboratory

Utah Teapot

Source: University of Utah, Utah Model Repository

Composed Scene

Source: Fausto Javier Da Rosa and Keenan Crane

Constant Width Bodies

Source: Keenan Crane

Crystals

Source: Mongze

Fox in the Wild

Source: Vlad Zaichyk

Horse and Heart

Source: Microsoft

RenderFormer Logo

Interior Room

Source: Microsoft

Shader Ball

Source: Wenzel Jakob

Tree

Source: elbenZ

Veach MIS

Source: Eric Veach

Videos

Check out extra video results including uncompressed videos and some reference videos.

Teaser Scenes

Dynamic demonstrations of RenderFormer's capabilities, showing object rotations, lighting changes, and material adjustments.

Cornell Box Roughness Adjustment

Bunny Roughness Adjustment

Tree Light Change

Tree Object Rotation

Fancy Scene Rotation

Composed Scene View Change

Animations

RenderFormer can render animations of scenes.

Cascade Cube Animation

Source: Tycho Magnetic Anomaly

Animated Crab

Source: Bohdan Lvov

Gyroscope Motion

Source: reddification

Animated Character

Source: mortaleiros

Marching Cubes Animation

Source: Tycho Magnetic Anomaly

Robot Animation

Source: Gouhadouken

Physical-Based Simulations

RenderFormer can render physically simulated scenes with complex dynamics and interactions.

Bowling Ball Physics Simulation

Source: SINOFWRATH

Rotating Box Dynamics

Constant Width Body Simulation

BibTeX

@inproceedings {zeng2025renderformer,
    title      = {RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illumination},
    author     = {Chong Zeng and Yue Dong and Pieter Peers and Hongzhi Wu and Xin Tong},
    booktitle  = {ACM SIGGRAPH 2025 Conference Papers},
    year       = {2025}
}