Google Integrates Gemini into Android, Enhancing In-App Contextual Awareness

Circle to Search is a feature built into the Android operating system that allows users to search for anything they see on their phone using a simple gesture. It integrates directly into the user experience, enabling users to perform searches without having to stop their current activity or switch apps. When a user circles a prompt or item they're curious about, Circle to Search provides information or step-by-step instructions for solving problems, such as physics and math word problems, directly from their phones and tablets.

Gemini on Android is an AI-powered assistant that helps users be more creative and productive by understanding the context of what's on their screen and the app they're using. With the upcoming update, users will be able to bring up Gemini's overlay on top of the app they're in, allowing for easy integration of Gemini's features. For instance, users can drag and drop generated images into emails or messages, or use 'Ask this video' to find specific information in a YouTube video. Gemini Advanced users will also have the 'Ask this PDF' option for quick answers without scrolling through documents. This update aims to provide more dynamic suggestions related to the content on the screen.

Gemini Nano with Multimodality is a new update for Android that will enable phones to process not just text input but also understand context from sights, sounds, and spoken language. This multimodal capability will first be introduced on Pixel devices and is designed to provide quick experiences while keeping information private. It will enhance features like TalkBack by providing richer and clearer descriptions for users who experience blindness or low vision, helping them understand images better. The multimodal capabilities also work offline, offering quick descriptions without a network connection.

A new feature being tested on Android uses Gemini Nano to provide real-time alerts during phone calls if it detects conversation patterns commonly associated with scams. Users will receive an alert if, for example, a caller posing as a bank representative makes unusual requests, such as urgent fund transfers or providing personal information like PINs or passwords. This feature is designed to protect users from fraud, with all the detection and alerting processes happening on-device to ensure the privacy of the conversation. More information about this opt-in feature will be shared later in the year.