Microsoft Open-Sources Phi-Ground 4B Model, Outperforms OpenAI Operator and Claude in Screen Clicking Accuracy

According to Beating, Microsoft recently open-sourced the Phi-Ground model family, designed to solve the problem of where AI should click on a computer screen. The 4-billion-parameter version, paired with larger language models for instruction planning, exceeded the clicking accuracy of OpenAI Operator and Claude Computer Use in the Showdown benchmark and ranked first among all sub-100-billion-parameter models across five evaluations including ScreenSpot-Pro.

The team trained on over 40 million data samples and found that three common training techniques used in academic papers became ineffective at scale. The key approach proved simple: output coordinates as regular numbers, such as “523, 417.” Previous research invented specialized position vocabularies for coordinates, but these failed to scale. The team also discovered that placing text instructions before images improved performance, as models could identify targets while processing pixels. Additionally, reinforcement learning methods like DPO improved accuracy even after fine-tuning.