Snapchat Introduces New AR Lenses, Platform Features, and Spectacles Capabilities

Snapchat has unveiled a series of new Lenses, platform features, and tools aimed at enhancing augmented reality (AR) experiences for developers and users alike. The updates are part of the Snap OS v5.58 release, which introduces innovative capabilities for Spectacles, Snap’s AR-enabled smart glasses, along with expanded tools for developers to create and experiment with AR applications.

New Lenses: Piano Tutor and Ball Game

Snapchat’s new Lenses demonstrate the power of the Spectacles camera and SnapML, Snap’s machine learning technology.

• Piano Tutor Lens: Designed to bring a personal piano instructor to users, this Lens uses custom machine learning models to identify and track a variety of pianos. It overlays notes onto the keys in real-time, helping users familiarize themselves with the keyboard layout and practice coordination. The Lens also evaluates performance, providing real-time feedback for continuous improvement.
• Ball Game Lens: This interactive Lens turns a physical ball into a controller for a series of challenges. A custom ML tracking model follows the ball’s movements, enabling users to interact with both physical and digital elements. It’s designed to help players refine their kicking skills while having fun.

Enhanced Platform Features and APIs

To empower developers to build more advanced AR experiences, Snap OS v5.58 includes several new platform features and APIs:

• Fetch API: Enables Lenses to interact with hosted multi-modal Large Language Models (LLMs), allowing for enhanced functionality and smarter interactions.
• Image Spatialization API: Uses generative AI to convert standard 2D image formats into realistic 3D visuals, adding depth and immersion to AR content.

Additional capabilities include:

• Spatial Anchors: Attach digital objects to specific locations within a Lens and revisit them later for consistent AR experiences.
• Basic Location: Access GPS coordinates of Spectacles to create outdoor experiences tied to real-world locations.
• Layout Mode: Import images from a smartphone via the Spectacles app and spatialize them into engaging 3D experiences.
• Web View: Drag and drop web pages directly into Lenses for added interactivity.
• Lens Unlock: Share a URL to unlock Lenses that are pending categorization or addition to Lens Explorer.

Get Started with Snap OS v5.58

To access the new features and start building with the updated tools, users and developers can update to the latest versions of Snap OS and the Spectacles app. Snap encourages developers to explore these capabilities and innovate with AR.

“We can’t wait to see what you create!” Snap said in its release, highlighting its commitment to supporting the AR community and driving the development of immersive, interactive experiences.

For more details and to update your devices, visit Snapchat’s Spectacles page.

Image: Snapchat