AI that clicks for you: Microsoft’s research points to the future of GUI automation

VentureBeat November 29, 2024
Michael Nuñez

A comprehensive new survey from Microsoft researchers and academic partners reveals that artificial intelligence agents powered by large language models (LLMs) are becoming increasingly capable of controlling graphical user interfaces (GUIs), potentially changing how humans interact with software.

The technology essentially gives AI systems the ability to see and manipulate computer interfaces just like humans do — clicking buttons, filling out forms, and navigating between applications. Rather than requiring users to learn complex software commands, these “GUI agents” can interpret natural language requests and automatically execute the necessary actions.

“These agents represent a paradigm shift, enabling users to perform intricate, multi-step tasks through simple conversational commands,” the researchers write. “Their applications span across web navigation, mobile app interactions, and desktop...

Today's Sponsors

Today's Sponsor