This paper details a large-scale empirical study conducted by Armature to evaluate how AI coding agents select third-party services and infrastructure when performing real-world software engineering tasks. The study addresses the lack of transparency in how LLM-based agents make architectural decisions, moving beyond simple benchmarks to analyze behavior within authentic development contexts.

Methodology

The research team employed a multi-layered, systematic approach to ensure statistical rigor and minimize bias:

1. Dataset Construction: The researchers analyzed thousands of public GitHub repositories to profile real-world engineering standards (languages, frameworks, deployment platforms). They then generated 75 synthetic repositories—complete with fake git histories, API keys, and verified dependency lockfiles—to match the distribution of real-world enterprise and startup stacks.
2. Experimental Framework: Experiments were executed within isolated, ephemeral sandboxes (E2B, Blaxel, and Daytona). A total of 16,893 runs were executed, with 5,292 sessions deemed valid based on strict criteria (e.g., successful implementation, lack of pre-existing bias in the codebase).
3. Human-in-the-Loop Simulation: To avoid the "in-house building" bias observed when agents have total autonomy, the researchers used an orchestrator (Gemini 3.7 Flash) to simulate human interactions. This forced the agent to justify tool selection and seek authorization, significantly impacting the decision-making process toward external solutions.
4. Evaluation: A separate instance of Gemini 3.7 Flash served as a judge, analyzing conversation logs and code diffs to extract the final tool choice. The validation criteria for a successful session included the requirement that the choice was not influenced by pre-existing code and that chosen observability tools were integrated into platforms rather than implemented in isolation.

Key Observations

• Agent Behavioral Divergence: The agents exhibit vastly different research patterns. Codex prioritizes targeted web searches (94% of sessions, often using site: operators), while Claude Code relies heavily on internal priors (searching only 30% of the time, though more deeply) and exhibits a higher tendency to build solutions in-house (19% vs. 10% for competitors).
• Contextual Sensitivity: The choice of infrastructure is highly dependent on the existing codebase language. For example, Resend dominates email service provider recommendations for TypeScript, while Postmark is favored in Go, and Sendgrid in Python.
• The "Mention vs. Choice" Gap: The data reveals a significant disconnect between tool awareness and tool adoption. Frequently mentioned frameworks like LangChain (194 mentions, 4 selections) or providers like PayPal (139 mentions, 0 selections) rarely translate to actual implementation, indicating that high visibility does not correlate with agent-driven adoption.
• Vendor Communication Impact: The research highlights that minor details on vendor pricing pages significantly influence agent logic. Negative outcomes were often triggered by features that appeared as "bloated" (e.g., Supabase losing to Neon because of bundled, unnecessary BaaS features) or restrictive constraints (e.g., Mailgun losing to Postmark due to "1-day retention" limits).
• Market Concentration: Some sectors show extreme consolidation. Stripe achieves a 90% success rate in the payment provider category, while others remain more competitive. Cloud platform-native solutions (S3, Azure) remain dominant in storage sectors, though simulated human intervention has proven capable of shifting the balance toward specialized competitors like Cloudflare R2.

The authors conclude that these 5.3k sessions provide a foundational dataset for understanding "agentic" software procurement, emphasizing that the way information is presented on a vendor's documentation page is a critical factor in determining whether an AI agent will select that service.