The Google artificial intelligence mode can now see and search images.
It's like Google Search, Lens, and Gemini have merged into a single tool.
Google is incorporating multimodal functionalities into its AI Mode chatbot, focused on search, which will allow it to "see" and answer questions about images. This advancement aims to expand access to AI Mode to "millions more" users.
The update combines a customized version of Gemini AI with Google's Lens image recognition technology. This will enable AI Mode Search users to take a photograph or upload an image and receive a "rich and complete response with links" regarding the content of the image. Starting today, this multimodal update is available on the Google app for both Android and iOS.
Robby Stein, Vice President of Product for Google Search, commented that "AI Mode builds on our years of work in visual search and takes it a step further." With the multimodal capabilities of Gemini, the AI mode can interpret the entirety of a scene in an image, analyzing how objects relate to each other, as well as their unique characteristics such as materials, colors, shapes, and arrangements.
Google has mentioned that this update utilizes a "fan-out technique" that formulates multiple queries about the image and the objects it contains, allowing for responses that are "incredibly nuanced and contextually relevant." This means it can identify books appearing in an image, suggest similar titles with good ratings, and answer questions to further refine the recommendations.
AI Mode for Search represents Google's response to Perplexity and ChatGPT Search, offering a chatbot-like experience that answers questions with AI-generated summaries drawn from Google's entire search index. Initially, AI Mode was launched exclusively for Google One AI Premium subscribers last month, although only within Labs. Now, Google has begun to make AI Mode accessible to "millions more" Labs users in the United States, beyond just paid subscribers.