Google's AMIE advances to expert-level audio-visual clinical consultations

Google Research has advanced its AMIE medical AI system to conduct real-time video consultations, achieving expert-level performance in a randomized controlled study with simulated consultations. The system, AMIE (Video), built on Gemini and Project Astra, addresses the limitations of text-based interfaces by perceiving non-verbal cues, guiding virtual physical examinations, and reasoning diagnostically in real time. To balance conversational latency with deep reasoning, AMIE (Video) uses an asynchronous multi-agent architecture with three specialized agents working in parallel. In a study with 100 scenarios, 300 live consultations, and 30 board-certified primary care physicians, AMIE (Video) was rated on par with PCPs on core clinical competencies. It was rated significantly higher than both PCPs and AMIE (Text) in eliciting physical signs and guiding virtual examinations, and patient actors preferred the video interface. Limitations include the use of simulated settings and occasional perceptual errors, and further validation with real patients is needed.