AI triage GitHub/Jira issues

An AI agent powered by an AI model connects to your GitHub repository and/or Jira instance and processes each new issue at the moment it is created. The system extracts meaning from the unstructured text of the title and description, classifies the ticket according to the team's internal taxonomy, and performs standard triage actions without engineer involvement.

What automation does in practice:

1. Picks up a new issue via webhook (GitHub Issues API or Jira Webhooks) immediately after creation.
2. Extracts key entities from the description: type (bug/feature/question), affected component or module, mentioned versions, environment, stacktrace.
3. Determines priority according to team rules: severity, affected users, business impact.
4. Sets labels and components in accordance with the approved taxonomy.
5. Searches for duplicates — semantic search across already open issues over the past 6-12 months.
6. Assigns an owner: by component, by module ownership, by round-robin within the team.
7. Leaves a comment with a concise structured summary for the engineer — what broke, where, how to reproduce, similar tickets.
8. Escalates to the Slack team channel if the issue is marked as critical or resembles a production incident.

Results from deployment practice: a senior engineer was spending 3 hours a week on manual triage — it became 20 minutes of quick review of edge cases. Time-to-label dropped from 18 hours to 2 hours. Duplicates are caught automatically — previously it took up to 1-2 days before someone noticed a repeat.

What automation does NOT do

Honest boundaries of AI agent responsibility:

• Does not make engineering decisions. Triage sets the labels and assigns an owner, but does not decide 'fix now or not' and 'which sprint' — that stays with the tech lead.
• Does not close issues. Even obvious duplicates are flagged and linked by the agent, but the final closure is done by a human — this is a safeguard against false matches.
• Does not work with private architecture without context. The agent needs a populated labels taxonomy, module ownership map, and examples of correctly labeled tickets.

Architecturally, the automation is built as a slim service between the issue tracker and the LLM: a webhook catches the event, context is assembled (issue text, similar open tickets, module ownership, previously labeled examples), the LLM returns a structured JSON with the classification, and the response is applied to the issue via the platform API. Everything is wrapped in retry logic and human-in-the-loop for contested cases.

Technical sequence

1. A webhook from GitHub or Jira arrives at the triage service (FastAPI or Node) on events issues.opened and issues.edited.
2. The service assembles context: title, description, author, existing labels, list of potential duplicates (top-5 by embedding similarity from the vector index).
3. The service builds a prompt for the AI model: the label taxonomy, module ownership map, few-shot examples of previously correctly labeled issues, and the text of the new ticket are passed in.
4. Claude returns JSON: { labels, priority, component, owner, duplicate_candidates, summary, confidence }.
5. The service validates the JSON against a schema: if the confidence level is below the threshold (e.g., 0.75) — it marks the issue with the tag needs-human-triage and does not assign an owner.
6. For confident cases, the service applies labels and the assignee via the GitHub or Jira API, and leaves a comment with a summary and links to similar tickets.
7. Critical issues are escalated to the team's Slack channel via an incoming webhook.
8. Every action is written to an audit log — tracking what the agent changed and why.

Solution components

Why custom-code, not ready-made no-code

For issue triage, three things matter that are poorly covered by ready-made builders: precise control over the prompt and few-shot examples (your taxonomy is specific), a duplicate vector index with persistent store embeddings, and a transparent audit log. All of this is handled in ~500-800 lines of Python or TypeScript with an LLM client, a vector DB (pgvector, Qdrant), and two API integrations. Typical stack: FastAPI + PostgreSQL with pgvector + AI model via Anthropic SDK + Octokit or Jira REST.

Human-in-the-loop as the default

The agent does not operate on full trust. Two mechanisms ensure safety:

1. Confidence threshold — when the level is below the threshold, the decision is marked needs-human-triage, labels are not applied.
2. Weekly review — once a week, the tech lead checks 20 random agent decisions against reality, few-shot examples are updated, and quality does not degrade.

Result: the majority of issues are labeled without engineer involvement, contested cases go to manual triage — but already with an agent-prepared summary and duplicate candidates, which still speeds up the work.

To start triage, the team needs ready data, access credentials, and minimal organizational preparation.

Data and access:

• GitHub Personal Access Token (scope repo) or a Jira API token with write permissions for labels, assignee, and comments.
• Approved label taxonomy — a list of types (bug/feature/task), priorities, and components. Usually 15-40 categories.
• Module ownership map: which person or team is responsible for which component or module.
• 100-200 correctly labeled issues from the past 6-12 months — for few-shot examples and building a vector index of duplicates.
• Anthropic API key for the AI model.

Infrastructure:

• A server or managed platform for the triage service (VPS, Render, Railway, AWS Lambda — any option will work).
• PostgreSQL with pgvector or Qdrant/Weaviate for the vector index.
• A Slack workspace and incoming webhook if escalation of critical issues is needed.

Team readiness:

• A tech lead or senior engineer as the automation owner — aligns the taxonomy and reviews quality for the first 2-3 weeks.
• 30-40 minutes per week from the owner for a weekly review of the agent's decisions in the first month, then 15 minutes.
• The team accepts the rules: duplicates are linked automatically, but a person closes them.

Implementation timeline:

For complexity «week» the expected timeline is 2-4 weeks. Week 1 — taxonomy alignment and few-shot dataset preparation. Week 2 — service implementation and webhook connection. Week 3 — shadow mode (the agent writes decisions to log but does not apply them), confidence threshold calibration. Week 4 — transition to production with human review.