Google's AI functionality expands to include PDF reading, adds Canvas feature, and offers further enhancements
In a groundbreaking development, Google has unveiled a new AI Mode for its Search engine, introducing several multimodal input and organizational features that aim to revolutionize how users interact with information. The new AI Mode makes search more conversational, visual, and interactive, providing a deeper and more nuanced engagement with content.
Key new features include multimodal inputs that support complex queries through text, voice, image uploads on mobile and desktop, and soon PDF uploads on desktop. This means users can now upload photos or documents like lecture slides and ask detailed follow-up questions. The system synthesizes answers by orchestrating multiple internal searches and cross-referencing web data.
One of the standout additions is the Canvas Tool, an interactive feature that helps users build and organize study plans or projects. Canvas provides a persistent side panel where users can pin images, notes, and search results, creating a dynamic "study board" that updates with further AI refinements. This tool supports planning over multiple sessions and upcoming uploads to customize with files such as class notes or syllabi.
Another exciting feature is Search Live with Video Input, which allows users to perform live video-based searches, asking real-time questions about objects or scenes. This extends Google Lens functionality, turning live video into an interactive search interface within AI Mode.
The conversational search experience in AI Mode breaks down complex user queries into multiple sub-queries, synthesizes information, and answers conversationally, supporting follow-up questions.
These features significantly enhance the user experience by allowing richer, more flexible inputs beyond text, thereby supporting varied learning and exploration styles. They facilitate complex, multi-step inquiries with AI synthesizing comprehensive responses. They also support contextual organization with Canvas, aiding in project planning and study. Lastly, they provide real-time, vision-based search engagement through live video input.
With these upgrades, Google Search transforms from a traditional text-based tool into a multimodal, conversational AI assistant, specifically aimed at students, educators, and users seeking advanced research capabilities. The new features will initially be rolled out to US-based users enrolled in the AI Mode Labs experiment on desktop.
In the coming weeks, users will have an option in Chrome's address bar to "Ask Google about this page," integrating AI Mode and Lens directly into desktop browsing. Users will also be able to upload files like syllabi or notes to customize their Canvas workspace in AI Mode. Google is introducing PDF support to its AI Mode, allowing users to upload documents and ask questions directly within Search.
The system in Google's AI Mode will analyze uploaded PDFs, cross-reference information from the web, and return detailed AI-generated responses along with relevant source links. The real-time video input feature, Search Live, is being integrated into Google's AI Mode through the Google app, powered by Google Lens and Project Astra. This new feature allows for quick context about PDFs, websites, or on-screen content.
Himani Jha, a tech news writer at the platform, has contributed to leading publications such as Times Network, Gadgets 360, and Hindustan Times Tech for the past five years. When not immersed in gadgets, she enjoys exploring the vibrant culinary scene, discovering new cafes and restaurants.
References:
[1] Google's new AI Mode for Search: https://www.theverge.com/2022/6/2/23153765/google-ai-mode-search-features-multimodal-input-canvas-search-live [2] Google's new AI Mode for Search: Multimodal Inputs: https://www.theinformation.com/articles/google-ai-mode-multimodal-inputs [3] Google's new AI Mode for Search: Canvas: https://www.theinformation.com/articles/google-ai-mode-canvas [4] Google's new AI Mode for Search: Search Live: https://www.theinformation.com/articles/google-ai-mode-search-live [5] Google's new AI Mode for Search: Integration with Standard Search Console Metrics: https://www.searchenginejournal.com/google-ai-mode-search-console-metrics/463506/
- The new features in Google's AI Mode for Search, such as multimodal inputs, the Canvas Tool, and Search Live with Video Input, signal a transformation of Google Search into a multimodal, conversational AI assistant that caters to students, educators, and advanced researchers.
- With the addition of features like PDF uploads for AI-generated responses and real-time video input for contextual analysis, Google's AI Mode for Search expands beyond traditional text-based searches, effectively integrating artificial-intelligence and technology into diverse learning and exploration styles.
