Abstract
Computer vision is a critical component in a wide range of real-world applications, including plant monitoring in agriculture and handwriting classification in digital systems. However, developing high-performance computer vision models traditionally demands both machine learning (ML) expertise and domain-specific knowledge, making the process costly, labor-intensive, and inaccessible to many. Large language model (LLM) agents have emerged as a promising solution to automate this workflow, but most existing methods share a common limitation: they attempt to optimize entire pipelines in a single step before evaluation, making it difficult to attribute improvements to specific changes. This lack of granularity leads to unstable optimization and slower convergence, limiting their effectiveness. To address this, we introduce Iterative Refinement, a novel strategy for LLM-driven ML pipeline design inspired by how human ML experts iteratively refine models, focusing on one component at a time rather than making sweeping changes all at once. By systematically updating individual components based on real training feedback, Iterative Refinement improves stability, interpretability, and overall model performance. We implement this strategy in IMPROVE, an end-to-end LLM agent framework for automating and optimizing object classification pipelines. Through extensive evaluations across datasets of varying sizes and domains, including standard benchmarks and Kaggle competition datasets, we demonstrate that Iterative Refinement enables IMPROVE to consistently achieve better performance over existing zero-shot LLM-based approaches. These findings establish Iterative Refinement as an effective new strategy for LLM-driven ML automation and position IMPROVE as an accessible solution for building high-quality computer vision models without requiring ML expertise.
Abstract (translated)
计算机视觉是农业植物监测和数字系统手写识别等众多现实世界应用中的关键组成部分。然而，开发高性能的计算机视觉模型通常需要机器学习（ML）专业知识和特定领域的知识，这使得这一过程成本高昂、劳动密集且对许多人来说难以企及。大型语言模型（LLM）代理作为自动化此工作流程的一种有前途的方法应运而生，但大多数现有的方法都存在一个共同的局限性：它们试图在评估前一次性优化整个管道，因此很难将改进归因于特定的变化。这种缺乏细节导致了不稳定的优化和较慢的收敛速度，从而限制了其有效性。 为了解决这个问题，我们引入了一种名为迭代细化的新策略，这是一种由LLM驱动的机器学习管道设计方法，灵感来自人类ML专家如何逐步完善模型——专注于一次改进一个组件而不是一次性做出重大改变。通过基于实际训练反馈系统地更新单个组件，迭代细化增强了稳定性、可解释性和整体模型性能。 我们在IMPROVE框架中实现了这一策略，这是一个端到端的LLM代理框架，用于自动化和优化对象分类管道。通过在不同大小和领域数据集上的广泛评估，包括标准基准测试和Kaggle竞赛数据集，我们证明了迭代细化使IMPROVE能够持续优于现有的零样本LLM方法。 这些发现确立了迭代细化作为LLM驱动的ML自动化的一种有效新策略，并将IMPROVE定位为无需机器学习专业知识即可构建高质量计算机视觉模型的一个可行解决方案。
URL
https://arxiv.org/abs/2502.18530
