Build your own dev agent

When to use this playbook

Use this guide when off-the-shelf copilots cannot integrate with your stack, or when you need strong governance, data residency, or custom workflows. Internal platform or productivity teams typically lead the effort.

Foundation requirements

• Approved access to an LLM (vendor or self-hosted) with clear spend controls.
• Secure secret management for repository tokens, CI credentials, and third-party APIs.
• Engineering workflows selected for automation with measurable success criteria.
• Stakeholders aligned on support ownership and escalation paths.

Core plays

1. Map the jobs-to-be-done. Interview developers to identify friction and prioritize automations (PR triage, test generation, deployment summaries). Document inputs, outputs, and human approval points.
2. Design the architecture. Choose runtime (serverless functions, containerized service, chat-based bot) that fits platform standards. Define integrations with Git, CI/CD, issue trackers, and knowledge bases.
3. Build the prompt and tool chain. Version system prompts, tool schemas, and guardrails in Git. Include evaluation harnesses and safety checks such as linting or test execution.
4. Implement governance. Log every interaction, store transcripts for audit, and enforce human approval before changes land in main branches. Provide dashboards for latency, success rate, and cost.
5. Run structured pilots. Launch with one or two volunteer teams, compare metrics to baseline, and iterate quickly on prompts and tooling. Publish learnings before onboarding additional teams.

Operating cadence

• Weekly triage of agent feedback, failure cases, and support tickets.
• Monthly cost and performance review with finance and security stakeholders.
• Quarterly roadmap discussion to prioritize new workflows or platform investments.

Success signals

• Target workflows show measurable time savings (20%+ reduction) without increasing change failure rate.
• Developers trust the agent because logs, guardrails, and support are transparent.
• Leadership has clear visibility into spend, usage, and planned enhancements.

Supporting assets

• Dev agent build checklist for day-to-day execution.
• FAQ covering security, operating model, and ROI questions.
• Background reading: manual/03-ai-agents/index.md.