On-call AI agent: diagnostics + auto-remediation PR

The AI agent works alongside the on-call engineer: reads alerts from Slack and the observability stack, collects diagnostic context, and prepares a pull request with a fix. It does not replace the on-call engineer — it responds to an incident first, so that by the time of escalation the context is already gathered and, in known cases, a fix is already proposed. In production mode, this saves the team 675 hours per month and closes 28 PRs without human involvement.

What the agent does

1. Listens to the on-call channel and monitoring webhooks — catches a new alert in seconds, not after the engineer opens the notification.
2. Extracts the stack trace, metrics, links to related dashboards, and recent deploys to build the full picture.
3. Searches for similar incidents in the history of Slack threads and runbooks — surfacing knowledge that typically lives only in the heads of experienced engineers.
4. Formulates a hypothesis about the incident cause and posts it to the thread as the first message, with a confidence level indicated.
5. If the incident matches a known pattern — opens a pull request with a fix and assigns reviewers.
6. Attaches evidence to the PR: logs, trace, links to similar cases, diff against previous fixes.
7. Stays in the thread and responds to the on-call engineer's follow-up questions until the incident is closed — a single source of truth instead of manually copying context.
8. After resolution, writes a short postmortem draft and records the new pattern for future incidents — the knowledge base is updated automatically.

The on-call engineer switches context less often: instead of the chain "alert → metrics → code → Slack → repository" they read a ready-made summary and make a decision. According to reference deployment data, 66% of the agent's suggestions receive positive feedback, and the cost of one interaction is $0,30.

What the agent does NOT do

• Does not merge a pull request without human approval — all changes go through standard code review and CI.
• Does not handle incidents for which there is no documented runbook or similar previous case — escalates to the on-call engineer with context already gathered.
• Does not make architectural decisions, does not refactor components, and does not touch code outside the permitted services — only targeted fixes for known patterns.

The agent is built on a multi-step orchestration pattern: LLM drives the cycle «observe → hypothesize → act → verify» until it finds a solution or decides to escalate. The core is a language model with tool use via an agent framework.

Architecture

The agent operates across three integration layers, each with its own tool calls:

Incident handling flow

1. Triggering event. An alert from the observability system lands in the on-call Slack channel. The webhook passes the event to the agent with a payload: severity, service, metric.
2. Context gathering. The agent makes a series of tool calls: reads the latest log lines, the metric chart for the past 24 hours, and deploy history for the last 6 hours.
3. Pattern search. The agent uses vector search across the Slack incident history and runbooks to find similar cases with their resolutions.
4. Hypothesis. The LLM formulates a hypothesis of the form «elevated latency on service X is caused by release Y — rollback or hotfix Z» with a confidence estimate.
5. Diagnosis post. The agent posts the first message to the thread: summary, hypothesis, links to evidence. The on-call engineer sees a summary, not raw logs.
6. Remediation path. If the pattern is known and confidence is high — the agent creates a branch, applies a fix from the template, opens a PR with a description, and assigns reviewers. If not — it stops and asks the on-call engineer to confirm the direction.
7. Human-in-the-loop. The on-call engineer reviews the PR, approves it or requests changes. The agent responds to comments: adds logs, revises the fix, explains the choice.
8. Post-mortem draft. After the incident, the agent compiles a timeline — what happened, what was done, how long it took — and posts the draft to the channel for editing.

How it is deployed on a project

1. Connecting observability: a webhook from Datadog, Grafana, New Relic, Sentry, or Prometheus Alertmanager to the agent service.
2. Repository integration: a GitHub App or GitLab access token with permissions to create branch, open PR, read commit history.
3. Installing the Slack bot in the on-call channel: reading events, posting responses, threading.
4. Importing historical incidents: parsing Slack threads and existing runbooks into a vector index — the core knowledge base of the agent.
5. Defining auto-remediation patterns: a list of incident types where the agent is permitted to open a PR (rollback deploy, changing a feature flag, bumping limits).
6. Guardrails: a list of services and repositories where the agent only reads, and a separate list where it can write.
7. Pilot: one week in «agent writes diagnostics only, no PRs» mode. The team evaluates hypothesis quality.
8. Expansion: after stable positive feedback, auto-remediation patterns are enabled one by one.

Where the value lies

The agent turns three pairs of hands into a single first responder who is always online. According to reference deployment data, 28 PRs per month are merged without human involvement — these are low-risk fixes that previously consumed senior engineers' time and pulled them away from their current work.

To launch an On-call AI agent, a team needs three readiness groups: access, historical data, and operational process. Without them, the pilot shifts to debugging integrations instead of real incident work.

Access and integrations

• Observability stack with webhooks: Datadog, Grafana, New Relic, Sentry, or Prometheus Alertmanager.
• Git repository with configured CI and code review (GitHub, GitLab, Bitbucket).
• Slack or equivalent with an on-call channel and bot installation rights.
• Technical agreement: read-only for most repositories, write (create branch + open PR) for the approved list.

Historical data

• Slack incident threads for the past 6–12 months — the more, the more accurate the pattern matching.
• Runbooks in any format (Confluence, Notion, markdown in the repository).
• A list of known auto-remediation patterns: which incident types the team is ready to delegate to the agent (rollback, feature-flag toggle, limit bump).

Team readiness

• On-call rotation is set up: duty engineer and escalation process in place.
• Code review is required for all PRs — the agent does not merge on its own.
• An owner is assigned: a senior SRE or tech lead who validates patterns and reviews false positives in the first weeks.

Implementation timeline

Complexity — medium. Full launch from contract to production — 6–10 weeks:

1. Weeks 1–2: integrations, access setup, incident history indexing.
2. Weeks 3–5: pilot in diagnostic mode, pattern configuration.
3. Weeks 6–8: enabling auto-remediation for one pattern, calibration.
4. Weeks 9–10: handover to the team and owner playbook.