Genie 3: A new frontier for world models
Genie 3’s capabilities include:
The following are recordings of real time interactions from Genie 3.
Modelling physical properties of the world
Experience natural phenomena like water and lighting, and complex environmental interactions.
Prompt: The video shows a first person perspective of someone navigating difficult terrain in the middle of a volcanic area. This is a real world video shot from the perspective of a wheeled robot that needs to traverse across a terrain. The vehicle has chunky offroad tires that crunch under the blackened rock. The camera is an egocentric camera mounted to the vehicle, and you can see the front tires just on the bottom of the camera along with the body of the robot. In the distance you can see smoke and lava flowing from the volcano. There are no other visible signs of life. There are lava pools that the agent is trying to avoid and random rock formations. The sky is a vivid blue.
Prompt: Jetski during the festival of lights
Prompt: Walking on a pavement in Florida next to a two-lane road from one side and the sea on the other, during an approaching hurricane, with strong wind and waves splashing over the road. There is a railing on the left of the agent, separating them from the sea. The road goes along the coast, with a short bridge visible in front of the agent. Waves are splashing over the railing and onto the road one after another. Palm trees are bending in the wind. There is heavy rain, and the agent is wearing a rain coat. Real world, first-person.
Prompt: Fast tracking real world video following a jellyfish swimming at high speed through the darkness of the deep sea between canyons covered in densely packed vent mussels with tiny white crabs crawling on them. Blurry hydrothermal vents in the distance spew thick, billowing plumes of vibrant blue, mineral-rich smoke from glowing rocky structures. Very dark, dim deep sea lighting, particles float in the cloudy ocean.
Prompt: A helicopter pilot carefully maneuvering over a coastal cliff with a small waterfall.
Simulating the natural world
Generate vibrant ecosystems, from animal behaviors to intricate plant life.
Prompt: Running by the shores of a glacial lake, exploring branching paths through the forest, crossing flowing mountain streams. Set amidst beautiful snow capped mountains and pine forest. Plentiful wildlife makes the journey a delight.
Prompt: Real world tracking shot swimming through deep dimly lit ocean between deep ocean canyons, densely packed vast school of jellyfish swimming, bioluminescent lighting.
Prompt: This is a natural, real-world landscape designed as a Japanese zen garden. The scene is set in the early morning under a clear sky. Soft, warm sunlight illuminates the garden, casting long, gentle shadows. The ground is covered in fine, white sand that is raked into meticulous swirling patterns. A small, still pond is present, with pink water lilies floating on its surface. Smooth, grey rocks of various sizes are placed throughout the garden, some with green moss on their surfaces. Key structures include a stacked stone cairn and a traditional Japanese stone lantern. The entire area is enclosed by a tall bamboo fence in the background. The visual style is photorealistic, with high detail in the textures of the sand, stone, and lush green vegetation.
Prompt: The environment is a natural, real-world landscape, specifically a dense arrangement of lush, vibrant foliage. The leaves are broad and deeply textured, displaying an array of green hues from emerald to lime, interspersed with hints of yellow and red, suggesting a rich, healthy ecosystem. Abstract dappled light filters down from above, creating shifting patterns of illumination and shadow across the leaves, highlighting their intricate veins and varied surfaces. The atmosphere is serene and deeply immersive, evoking a sense of being within a vibrant, living natural world. Small water droplets are visible on some leaf surfaces, reflecting the ambient light. The background is a soft blur of similar foliage, emphasizing the foreground elements. The air appears humid and still.
Modelling animation and fiction
Tap into imagination, creating fantastical scenarios and expressive animated characters.
Prompt: A vibrant 3D style, an adorable, fluffy creature bounding across a vibrant rainbow bridge in a fantastical landscape. The creature is small and compact, with fur that mimics the warm hues of a sunrise – oranges, yellows, and pinks blending seamlessly together. Its most striking feature is a pair of large, perked ears, shaped like those of a German Shepherd, adding a touch of playful contrast to its otherwise rounded form. As it runs on four short legs across the rainbow, its fur appears to ripple and flow, adding to its sense of dynamism and energy. The rainbow bridge arches gracefully through a whimsical landscape, perhaps filled with floating islands, glowing flora, and swirling clouds. The lighting is bright and cheerful, casting a warm glow on the creature and its surroundings. The overall impression is one of joy, wonder, and boundless energy, capturing the creature's playful spirit and the magical nature of the world it inhabits. This image evokes a sense of childlike whimsy and invites the viewer to imagine the adventures that await this charming creature in its fantastical realm.
Prompt: Being a lizard, origami style
Prompt: A fantastical, wide-angle shot captures a lush, enchanted forest bathed in the soft glow of twilight. The player controls a large firefly flying through towering trees with vibrant foliage creating a dense canopy overhead, filtering the sunlight and casting dappled shadows on the forest floor. Nestled among the branches are a handful of charming tree houses, each glowing with a warm, inviting light. The tree houses vary in size and design, some resembling whimsical castles, others cozy cabins. Tiny details, like glowing windows and miniature balconies, add to their charm. A winding path, barely visible beneath the undergrowth, leads the viewer's eye deeper into the enchanted forest. The overall scene evokes a sense of wonder, tranquility, and the magic of childhood dreams.
Prompt: A serene Irish landscape, with rolling emerald-green hills, misty lakes, and rugged mountains, suddenly trembles violently—as if the earth itself is being torn apart. In a moment of surreal chaos, entire sections of land rip free, rising into the sky in jagged, brutalist formations, their rocky undersides exposed like raw, fractured earth. The lakes are wrenched upward, now suspended in the sky, their waters spilling downward in colossal waterfalls, creating an apocalyptic storm of mist and rain over the land below. The camera pulls back, revealing a new impossible geography—mountains floating, cliffs inverted, rivers twisting mid-air—as gravity itself bends, turning the once-peaceful countryside into a brutalist, surreal monument to nature’s violent transformation.
Exploring locations and historical settings
Transcend geographical and temporal boundaries to explore places and past eras.
Prompt: A real world mountainous environment in the Alps. The landscape features steep, rocky cliffs and narrow gorges filled with loose scree and debris. The rock is predominantly grey and white, with patches of green vegetation clinging to the cliff faces. The top of the gorge opens up to a vista of dense evergreen forests and meadows. The overall theme is one of rugged, natural beauty and extreme terrain.
Prompt: Venice by Vaporetto. The canals of Venice are recreated with painstaking detail. The water has realistic reflections and wakes. The buildings show crumbling plaster and centuries of weathering. The scene is populated with other gondolas, water taxis, and barges.
Prompt: Exploring the palace of Knossos on Crete as it would have stood in its glorious heyday.
Prompt: Walking around on a beautiful day out in Hinsdale, Illinois. Real world. There are cars parked. The person filming is standing on the sidewalk, there are flocks of birds flying overhead.
Prompt: A biking enthusiast driving on a narrow road on an edge of a cliff in India, the Killar-Kishtwar Road. Real-world, first-person, only hands on handles visible.
Pushing the frontier of real-time capabilities
Achieving a high degree of controllability and real-time interactivity in Genie 3 required significant technical breakthroughs. During the auto-regressive generation of each frame, the model has to take into account the previously generated trajectory that grows with time. For example, if the user is revisiting a location after a minute, the model has to refer back to the relevant information from a minute ago. To achieve real-time interactivity, this computation must happen multiple times per second in response to new user inputs as they arrive.
Environmental consistency over a long horizon
In order for AI generated worlds to be immersive, they have to stay physically consistent over long horizons. However, generating an environment auto-regressively is generally a harder technical problem than generating an entire video, since inaccuracies tend to accumulate over time. Despite the challenge, Genie 3 environments remain largely consistent for several minutes, with visual memory extending as far back as one minute ago.
Prompt: POV action camera of a tan house being painted by a first person agent with a paint roller
Prompt: A Victorian street with a grey house. The grey house has a portal ringed by magical sparks. The portal leads to a vast desert filled with dunes, and that desert is visible from the outside. The agent can walk into the portal and is teleported to the desert.
Prompt: A classroom where on the blackboard at the front of the room it says GENIE-3 MEMORY TEST and underneath is a beautiful chalk picture of an apple, a mug of coffee, and a tree. The classroom is empty except for this. Outside the window are trees and a few cars driving past.
Prompt: This is a fantastical, whimsical forest environment. The lighting is bright and cheerful, suggesting a sunny day with dappled light filtering through a dense canopy of lush, oversized leaves. The air is clear and still. The ground is a soft, verdant carpet of moss and unusually large, brightly coloured mushrooms in shades of red and blue, their caps dotted with white. Winding dirt paths, well-trodden and narrow, weave between towering, ancient trees with smooth, grey bark. Interspersed throughout the forest are charming, mushroom-shaped houses, with intricate wooden doors and tiny, circular windows, each one unique in its design and colour palette, ranging from vibrant reds to gentle blues and greens. Various small, friendly forest creatures, such as colourful butterflies and tiny singing birds, flit amongst the foliage, adding to the lively atmosphere. There is an abundance of peculiar, oversized flowers blooming in an array of pastel and bright hues, releasing a gentle glow.
Prompt: An extremely enormous, realistic gorilla, draped in a flamboyant, emerald red vest with ornate brass buttons and an elaborate, feathered bicorne hat, brandishing only a vintage silk parasol, navigates a series of outrageously extravagant, moss-laden McMansions where grand marble structures are subtly embraced by sprawling, ancient rose bushes and creeping ivy.
Prompt: Walking around ancient Athens, Greek architecture, marble
Genie 3’s consistency is an emergent capability. Other methods such as NeRFs and Gaussian Splatting also allow consistent navigable 3D environments, but depend on the provision of an explicit 3D representation. By contrast, worlds generated by Genie 3 are far more dynamic and rich because they’re created frame by frame based on the world description and actions by the user.
Promptable world events
In addition to navigational inputs, Genie 3 also enables a more expressive form of text-based interaction, which we refer to as promptable world events.
Promptable world events make it possible to change the generated world, like altering weather conditions or introducing new objects and characters, enhancing the experience from navigation controls.
This ability also increases the breadth of counterfactual, or “what if” scenarios, that can be used by agents learning from experience to handle unexpected situations.
Choose a world setting. Then, pick an event, and see Genie 3 create it.
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0
