Visko's Orbis Achieves Real-Time Streaming Breakthrough on Google Cloud
Discover how Visko's groundbreaking Live World Model, Orbis, revolutionizes real-time video generation by streaming content that evolves with user inputs, all powered by Google Cloud TPUs.
Key Facts
- Visko's Orbis achieves 1.07x playback speed, showcasing real-time AI's potential in media.
- 30-day deployment on Google Cloud TPUs highlights agility in tech partnerships for startups.
- Live streaming demands strict latency management, revealing vulnerabilities in traditional models.
- Transitioning to native JAX boosted audio processing speed by 5.8x, enhancing overall performance.
- Visko's model opens new markets in interactive media, indicating a strategic shift in content creation.
Summary
Summary
Visko, an AI startup, faced the challenge of deploying a real-time streaming world model, Orbis, that could continuously generate video and audio without interruption. By collaborating with Google Cloud, they implemented an end-to-end live inference pipeline on Cloud TPUs, achieving real-time rendering at 1.07x playback speed within 30 days.
Background
Visko operates in the generative AI industry, focusing on interactive world models. Before deploying their solution, they were limited by traditional offline video generation methods, which could not meet the demands of real-time interactivity and responsiveness required for their flagship product, Orbis.
Challenge
The primary challenge was to create a live world model that could stream video and audio continuously without delays. Each chunk of video needed to be rendered and synchronized with audio faster than real-time playback, which posed significant technical hurdles compared to standard offline video generation.
Solution
Visko and Google Cloud engineers developed a multimodal pipeline that combined video generation, super-resolution upscaling, and synchronized audio processing on a single Cloud TPU slice. This setup allowed for rapid processing of viewer prompts, enabling real-time updates to the video stream without breaking continuity.
Results
The implementation resulted in a successful streaming performance, achieving rendering at 1.07x playback speed. The entire deployment process, from initial collaboration to production handover, was completed in just 30 days.
Key Insights
- Real-time video streaming requires a different approach than offline generation, focusing on minimizing latency for each individual viewer session.
- Leveraging existing frameworks like JAX and torchax can accelerate the transition to high-performance environments like Cloud TPUs.
- Optimizing each stage of the pipeline is crucial for maintaining quality and performance under strict latency requirements.
Customer Testimonial
"Visko is bringing an exciting new class of real-time, steerable workloads to the Cloud TPU ecosystem. By co-engineering their live multimodal pipeline on a single TPU slice using JAX, torchax, and custom Pallas kernels, we proved how quickly a lean startup team can go from PyTorch prototype to render at 1.07x playback speed on Google Cloud's AI Hypercomputer." – Dave Elliott, Head of Developer Advocacy for AI, Google Cloud
Entities Mentioned
Companies
Products
Technologies
People
Key Concepts
Definitions
- Live World Model
- A video world model that generates continuously and responds in real-time to user inputs without breaking continuity.
- Cloud TPU
- Google's Tensor Processing Unit designed for accelerating machine learning workloads.
- JAX
- A high-performance Python library for numerical computing that allows for automatic differentiation and compilation.
- torchax
- An interoperability layer that enables running PyTorch models on Cloud TPUs via JAX.
- Pallas
- A custom kernel programming language for TPUs used in conjunction with JAX.
Use Cases
- →interactive entertainment
- →live commerce
- →AI companions
- →immersive education
- →robotics simulation
Frequently Asked Questions
What is Visko's main product?
Visko's flagship product is Orbis, a live world model that streams video based on user prompts, allowing for real-time interaction and scene evolution.
How does Visko ensure low latency in its streaming?
Visko achieves low latency by optimizing its multimodal pipeline to run on a single Cloud TPU slice, ensuring that each video chunk is processed faster than real-time playback.
What technologies did Visko use for their live pipeline?
Visko's live pipeline utilizes JAX, torchax, and Pallas, which work together to optimize performance and enable the efficient processing of video and audio in real-time.
What challenges do live world models face compared to offline video generation?
Live world models must meet strict real-time deadlines, as each video chunk must be rendered before the viewer finishes watching the previous one, unlike offline generators where latency can be managed.
How can startups benefit from Visko's experience with Cloud TPU?
Startups can learn from Visko's approach to building low-latency generative models on Cloud TPU by utilizing the same tools and techniques, such as optimizing their pipelines and leveraging Google Cloud's resources.