Application Refactoring · IT Operations Automation
Factory needed secure observability and a faster feedback loop for autonomous SDLC agents running inside enterprise environments with strict privacy controls, custom LLM tooling, and limited tolerance for opaque debugging workflows.
Reported impact
Factory doubled iteration speed for prompt and agent improvement, reduced open-to-merge time and code churn for customers, and established a secure, enterprise-friendly observability stack for autonomous engineering workflows.
Factory runs AI Droids that automate stages of the software development lifecycle and uses self-hosted LangSmith as the observability backbone inside tightly controlled customer environments. LangSmith traces are exported into AWS CloudWatch so engineers can correlate agent events with infrastructure logs and follow data through each step of a multi-stage LLM pipeline without building a proprietary logging stack. Feedback is attached directly to individual LLM calls through the Feedback API, then exported into datasets for analysis. Factory’s engineers use custom LangChain tooling to compare good and bad code-review comments, hypothesize why prompts failed, re-prompt the model, and improve accuracy. This closed-loop design lets the team keep enterprise data private while continuously tuning Code Droid and related Droids from production evidence rather than ad hoc manual review.
Droids execute SDLC tasks, Self-hosted LangSmith records each LLM step and exports traces to AWS CloudWatch, engineers attach workflow feedback through the Feedback API, and custom LangChain tooling turns those labeled examples into prompt updates and new runs that can be evaluated again.
Factory integrated LangSmith to export traces to AWS CloudWatch logs
Factory was able to 2x their iteration speed.
With feedback available next to every LLM call
a 3x reduction in code churn on code impacted by Droids in the first 90 days
a ~20% reduction in open-to-merge time
Review how this reference was scored and trace each normalized claim back to its documented source.
Scoring methodology
Customer Reference Value measures usefulness as a customer reference. Evidence Confidence measures the quality and traceability of available evidence, not independent verification.
Traceability
Field coverage shows which facts each source supports. Dates are shown only when documented by the source.
The company profile states 'Operating headquarters' in 'San Francisco, CA, US' and 'Factory has 29 employees as of Feb 4, 2024'.
Related
OpenAI Sample Oct 9, 2026 Asana Asana needed to cut the cost and latency of its StackAI browser agent so it could offer customers more capable models for browser-driven workflows without unsustainable operating expense. $0.47 average estimated model cost with the optimized GPT-6.1 Sol workflow Codex · Command · GPT-6 Astra OpenAI Sample Oct 8, 2026 Braintrust Braintrust wanted to respond to customer feature requests faster and run more engineering experiments, but slower coding tools and backlog-driven workflows made it costly to turn feedback into working prototypes in real… 50% of the Braintrust team moved to Codex in one month Codex · GPT-5.5 OpenAI Sample Sep 23, 2026 Cognition Cognition is the company behind Devin, an autonomous software engineer used by businesses around the world, from big banks to tech-native startups. As Cognition’s engineering teams write more code, reviewing that work… Cognition is applying GPT‑6 Astra across its product lineup. "We’ve been using it to make improvements across… GPT-6 Astra 
