Building a Simple AI Support Ticket Triage Workflow

A small SaaS team I helped last quarter was getting around 80 support tickets a day through a shared inbox, split across three people who also had other jobs. Nothing about the volume was unmanageable on its own — the problem was that urgent tickets (a paying customer locked out, a billing error, a broken integration) sat in the same queue as "how do I change my timezone" for hours at a time, because nobody was triaging, just answering in whatever order tickets arrived. This is a walkthrough of the triage workflow we built, what the prompts actually looked like, and the two ways it broke that we had to design around rather than ignore.

What the workflow actually does

It's two steps, not one, and keeping them separate turned out to matter:

1. Classify — the incoming ticket gets tagged with an urgency level and a category (billing, bug report, how-to question, feature request, account access) before a human ever opens it.
2. Draft — for categories where a template-ish response is often appropriate (how-to questions, common billing questions), the model also writes a first-pass reply. This reply is never sent automatically. It sits as a draft attached to the ticket for a human to edit or discard.

The instinct a lot of teams have is to build one prompt that classifies and replies in a single pass. We tried that first and it produced worse classifications — asking the model to also compose a reply seemed to bias it toward under-flagging urgency, because a friendly draft reply "feels" like the ticket is being handled, even when the underlying issue is actually serious. Splitting it into two separate calls, each with a narrow job, fixed that.

The classification prompt

Here's roughly what the classification step looks like, trimmed down from the version actually in use:

You are triaging a customer support ticket. Read the ticket text and
output a JSON object with exactly these fields:

- urgency: one of "critical", "high", "normal", "low"
- category: one of "billing", "bug", "how_to", "feature_request",
"account_access", "other"
- reason: one sentence explaining the urgency choice
- confidence: "high" or "low" — use "low" if the ticket is ambiguous,
sarcastic, or missing information needed to classify it confidently

Guidance:
- "critical" means the customer cannot access something they are
paying for, or money was charged incorrectly.
- Vague frustration ("this is broken", "not working") without
specifics is "normal" urgency and "low" confidence, not "critical."
- If the ticket mentions legal action, cancellation, or public
complaint threats, set urgency to "high" regardless of category.

Ticket:
"""
{ticket_text}
"""

Output only the JSON object, no other text.

Two things in there earned their place through trial and error, not because they seemed like good ideas upfront. The confidence field exists because early versions of this prompt classified ambiguous tickets with the same false certainty as clear ones — a one-line "it's broken" ticket got the same crisp "normal / bug / high confidence" output as a detailed bug report with logs attached. Adding an explicit instruction to flag low confidence, and then routing low-confidence tickets to a human queue first rather than auto-sorting them, cut down on a specific failure: tickets that were actually urgent but written vaguely by a frustrated customer kept landing in "normal."

The explicit definition of "critical" also mattered more than expected. Without it, the model's idea of critical drifted toward anything written with exclamation points or in all caps — tone, not substance. Anchoring the definition to something concrete (paying customer locked out, incorrect charge) made the urgency calls track actual business impact instead of how upset the customer sounded.

The draft-reply prompt, and why the review step is not optional

For tickets classified as how_to or common billing questions, a second prompt drafts a reply using the ticket text, the classification output, and a short set of canned facts (refund policy, plan limits) fed in as context. The reply is written in a specific tone — direct, not overly apologetic, no "I understand how frustrating this must be" boilerplate stacked on every message — because that phrasing had shown up so often it started reading as obviously templated to customers who received two or three replies over an exchange.

What actually happened the first week we turned drafting on: a customer asked why their trial hadn't converted to a paid plan, and the draft reply confidently explained a billing cycle rule that didn't match how our actual billing worked — it was a plausible-sounding explanation stitched together from the general shape of SaaS billing rather than our specific policy, because the reference material we'd fed it hadn't covered that exact scenario. It read completely reasonable. If it had gone out unreviewed, we'd have told a customer something false about their own bill.

That's the concrete reason the review step isn't a "best practice" nicety here, it's load-bearing: every draft goes to a human before anything is sent, and the person reviewing is told explicitly to check any factual claim in the draft against the actual account or policy, not just proofread the tone. We also started keeping a short log of edits reviewers made to drafts, which turned out to be a useful signal — if a specific claim gets corrected repeatedly, it means the reference context is missing something, not that the model is randomly unreliable.

If a draft reads a little stiff or robotic and a reviewer wants to loosen it up before sending without rewriting from scratch, running it through something like the AI Content Humanizer before it goes out is a faster fix than a manual rewrite, though it's still the reviewer's job to check the underlying facts are right first — that tool changes phrasing, not accuracy.

Where it breaks down

Two categories of tickets consistently gave the workflow trouble, and neither is fixable by tweaking the prompt further — they needed a process change instead.

• Ambiguous tickets that mix categories. "I was charged twice and also can't log in" is both billing and account access, both plausibly urgent, and the model would pick one category and under-weight the other. We added a rule that any ticket triggering two categories at once gets bumped to high urgency and routed to a human first, rather than trying to get the classifier to output multiple categories cleanly.
• Tone mismatches on sensitive tickets. A ticket from a customer whose account was suspended for suspected fraud, written angrily, got a draft reply in our standard friendly tone that read as tone-deaf given the situation — technically accurate information, wrong register entirely. We excluded anything touching account suspension, security, or legal threats from auto-drafting altogether; those go straight to a human with no draft attached, because getting the facts right isn't enough when the tone itself is the thing that needs the most judgment.

Setting this up without writing custom glue code

You don't need a bespoke pipeline to test whether an approach like this is worth building out. Before we wrote any integration code, we sketched the whole flow — ticket comes in, gets classified, routes to the right queue, drafts get generated for specific categories only, everything human-reviewed before sending — as a plain-English description and ran it through the AI Automation Builder to get a structured workflow plan we could hand to whoever was actually wiring up the ticketing system's webhooks. It's a useful sanity check before you commit engineering time: if the automation idea doesn't hold together as a clear step-by-step plan, it's not going to hold together as code either.

The other thing worth deciding upfront, before any of this touches real customers, is what "good enough to ship" looks like for the classification step specifically — see the prompt patterns post for how to structure that kind of testable, format-constrained prompt so you're not guessing at whether a change made things better or worse.

Handling volume spikes without breaking triage

The workflow held up fine at 80 tickets a day. It behaved differently during a payment provider outage that generated close to 300 tickets in four hours, almost all billing-related and almost all genuinely urgent at the same time. Two things broke that hadn't shown up at normal volume.

First, "critical" stopped being a useful filter — when every ticket in the queue is critical, urgency ranking collapses back into arrival order, which is exactly the problem triage was supposed to solve. We added a second-pass rule for exactly this situation: if more than a threshold percentage of open tickets in a rolling window are classified critical, the system also groups them by root cause similarity (using the ticket text) rather than just urgency, so a human can see "47 of these are the same payment outage" instead of working through 47 individually-critical tickets one at a time believing each is a distinct incident.

Second, draft-reply generation for the how-to category briefly used up API rate limit headroom that billing-related classification calls also needed, which meant classification itself was queuing behind draft generation for non-urgent tickets. We split them onto separate rate-limit budgets so a burst of low-priority draft requests can never delay the classification step for a ticket that might turn out to be critical. Neither fix was obvious until the outage actually happened; both are the kind of thing worth deciding on paper before go-live rather than during the next real incident.

What changed after three months

The measurable outcome wasn't "AI answers tickets now" — it's that critical and high-urgency tickets get seen within minutes instead of sitting in inbox order, and the how-to category (which was maybe a third of total volume) went from a full write-from-scratch reply to an edit-and-send in most cases. The team still reads and sends every message themselves. What changed is where their attention goes first, and that the drafts they're editing are usually closer to the right answer than a blank reply box — not always, which is exactly why the review step stays mandatory rather than becoming a formality.