Apple enhances Siri's 'on-screen awareness,' allowing it to interact with your device.
Siri, smarter and more screen-aware, could finally understand what you're thinking, at least in part.
Apple is working on a new feature called "screen awareness," which will allow Siri to recognize and interact with the content displayed on the user's screen. This innovation will be accompanied by APIs so that third-party developers can incorporate this feature into their applications. Currently, Apple is conducting tests to integrate ChatGPT, which will enable Siri to answer questions based on screenshots.
Although "screen awareness" will not be available in the iOS 18.2 beta, it is expected to launch alongside iOS 18.4, scheduled for 2025. When compared to other digital assistants, Siri has performed satisfactorily, especially in comparison to Microsoft's Cortana, and now Apple seeks to enhance its understanding of what the user is viewing.
This feature was presented preliminarily in June 2024, and Apple’s developer documentation provides more details about its development. The concept behind "screen awareness" is relatively simple: if the user is reviewing a document or a webpage and has questions about what they are seeing, they can ask Siri, and it should provide a relevant answer or perform requested actions, such as sending content to a third-party app.
If "screen awareness" works as expected, Siri could become more intuitive and not require such elaborate descriptions to execute tasks. For example, a user could request a summary of a document without having to provide the full text.
Additionally, it has been reported that Apple will provide APIs to developers so they can make the content of their applications accessible to Siri. The goal is for developers to have this tool well ahead of the official launch, allowing users to utilize "screen awareness" in third-party applications from day one of its availability.
Regarding the integration of ChatGPT, it is being tested in the iOS 18.2 beta, where Siri could use screenshots to answer questions about visual elements, such as images, PDFs, or videos. However, "screen awareness" is a step further, as this feature is expected to interact directly with what is displayed on the screen.
If realized, Siri could become an even more useful digital assistant. Expectations are high, and some believe this could place Siri in a competitive position against other AI assistants in the current market.
