Microsoft Copilot Vision introduces a new way of interacting with computers by allowing the AI to see what users see and engage with them in real-time. It understands both text and images on web pages, can answer questions, suggest next steps, and assist without disrupting workflow. For example, it can help users search for furniture, choose color palettes, and suggest arrangements when furnishing a new apartment.

Rollout and Availability

Copilot Vision will soon be available to a limited number of Copilot Pro subscribers in the United States through Copilot Labs. This phased rollout aims to gather feedback and improve the product before a wider release. Copilot Labs serves as a testing ground for advanced tools, offering a glimpse into upcoming features.

The first feature in Copilot Labs, Think Deeper, enhances Copilot's ability to solve complex problems using advanced reasoning models. It provides detailed, step-by-step answers and is currently available to select Copilot Pro users in several countries, including the United States, United Kingdom, Canada, Australia, and New Zealand.

Copilot Vision Capabilities

Copilot Vision addresses a limitation in the Copilot experience by allowing the AI to understand the context of what users are doing or viewing. Integrated within Microsoft Edge, it can understand web pages, answer questions, suggest next steps, and assist with tasks through natural language interaction. This feature is entirely opt-in, with no content stored or used for training, and it respects machine-readable controls on AI for pre-approved websites.

Copilot Vision is designed with user and creator interests in mind. It is blocked on paywalls and sensitive content, and it does not engage directly with the web but rather answers questions. The service aims to drive traffic to websites responsibly, and feedback from users will be crucial in refining the feature.

Copilot Vision represents a significant advancement in AI interaction, balancing pioneering features with a deep sense of responsibility. Initially available to a subset of Pro users in the United States, it aims to provide useful and rich experiences while ensuring safety and user control.