The Unsolved Challenges of LLMs in Open-Ended Web Tasks: A Case Study

We investigate the challenges associated with developing goal-driven AI agents capable of performing open-ended tasks in a web environment using zero-shot learning. Our primary focus is on harnessing the capabilities of large language models (LLMs) in the context of web navigation through HTML-based user interfaces. We evaluate the MiniWoB benchmark and show that it's a suitable yet challenging platform for assessing an agent's ability to comprehend and solve tasks without prior human demonstrations. Our main contribution encompasses a set of extensive experiments where we compare and contrast various agent design considerations, such as action space, observation space, and the choice of LLM, with the aim of shedding light on the bottlenecks and limitations of LLM-based zero-shot learning in this domain, in order to foster research in this area.