Google Lens now allows users to conduct searches using videos.

Google Lens has introduced a new feature that allows users to conduct searches through video. Now, if you can't capture what you want with just an image, you can record a video and ask questions using your voice about what you're observing. This feature generates an AI-powered Overview and search results, all based on the content of the video and your query. The functionality is rolling out today in Search Labs on both Android and iOS.

During the I/O event in May, Google first showcased this ability to search via video. For instance, if someone is at an aquarium and is curious about the fish they're seeing, they can point their phone at the tank, open the Google Lens app, and hold down the shutter button to start recording. While recording, they can ask: "Why are they swimming together?" Google Lens uses the Gemini AI model to provide an appropriate answer.

Rajan Patel, Vice President of Engineering at Google, explained that the video is captured "as a series of image frames, and then computer vision techniques" that were already being used in Lens are applied. However, they have taken the process further by feeding the information into a "custom" Gemini model designed to understand multiple frames sequentially and provide answers based on web information.

Although sound in a video, like a bird's song, cannot currently be identified, Patel mentioned that this is an area where Google is experimenting. Additionally, Google Lens is revamping its photo search feature, allowing users to ask questions more directly using their voice. To use this option, you simply focus the camera on the object, hold down the shutter button, and ask. Previously, users could only type their queries after taking a photo. This voice question feature will be rolled out globally on Android and iOS, although for now, it is only available in English.