Matrix-3D: From Visual Generation to 3D Construction, Driving the Virtual Future

As video generation evolves beyond static frames and short clips, it is stepping into a new stage with the creation of fully navigable virtual worlds powered by AI spatial intelligence.

Leading this transformation, the Matrix large-scale foundation model, officially launched on August 12, 2025, by Skywork, a leading AI company in the industry, introduces two groundbreaking sub-models: Matrix-3D and Matrix-Game 2.0. Together, they deliver a complete pipeline that transforms 2D visual inputs into structurally coherent, high-fidelity, fully explorable 3D environments. This advancement lays the foundation for progress in virtual reality, robotic navigation, and intelligent agents. Matrix-3D, in particular, raises the bar for panoramic 3D scene generation and fully roamable exploration.

Technological Core of Matrix-3D

Matrix-3D is built on three key modules that overcome long-standing challenges in panoramic and 3D scene construction, achieving unlimited viewpoints with geometric and visual consistency.

Trajectory-Guided Panoramic Video Generation utilizes scene mesh renderings as conditional inputs with camera trajectory guidance to align generated panoramas, reducing occlusion errors and structural distortions.

Speed-Quality Balance 3D Reconstruction converts generated videos into navigable 3D spaces via two modes: a feed-forward large panorama reconstruction model for rapid 3D scene reconstruction, and an optimization-based pipeline for accurate and detailed 3D scene reconstruction.

Matrix-Pano Dataset offers more than 116,000 panoramic video sequences with camera trajectories, depth maps, and text annotations, enabling the model to capture geometric detail, lighting variations, and occlusion patterns for highly realistic 3D output.

Performance Breakthroughs

In benchmark tests against commonly used Panoramic Video Generation Models, including 360DVD, Imagine360, and GenEx, Matrix-3D demonstrates the highest PSNR and SSIM scores, highlighting its precise geometric fidelity and structural consistency, while achieving markedly lower LPIPS and FID scores, signifying more realistic output.

Enhanced with a LoRA fine-tuning strategy, Matrix-3D optimizes minimal parameters without compromising stability. This enables rapid, controllable reconstruction with stable occlusion handling, consistent shading, seamless 360-degree navigation, extended spatial range, and features that previously required complex rendering pipelines.

Real-World Applications

Matrix-3D's visual quality and spatial precision allow fine-grained control over 3D structure and semantics. It's more than a tool for creating appealing images; it is a content engine for industries that rely on virtual spaces while maintaining cost efficiency and scalability.

VR/AR: Rapid creation of realistic virtual worlds, enabling dynamic exploration and immersive experiences.

Simulation and Robotics: Create controllable simulation environments with real-world geometry constraints for robotics training and autonomous AI testing.

Film and Gaming: Generating cinematic scenes or game levels from concept art to support virtual production while significantly reducing asset creation time.

Vision for the Future

Skywork plans to release an open-source subset of curated 3D scenes and panoramic video datasets to encourage collaboration in academia and industry. This will support research in real-time rendering, adaptive environment generation, and high-fidelity virtual content pipelines.

A Leap Forward: Looking Ahead

The debut of Matrix-3D marks the beginning of a new era in AI-driven worldbuilding. It combines panoramic scope, precise structural control, and real-time performance to push the limits of spatial intelligence. As an important milestone toward embodied intelligence and AGI, Skywork will build on Matrix-3D to pioneer a future shaped by unprecedented virtual innovations.