Choosing an engagement
Managed support vs AI operations
Is managed support enough for a live AI system, or does it also need AI operations? Ask whether anyone would notice if the system kept running but started giving worse answers. Managed support catches what breaks loudly: outages, errors, slow requests, failed releases, and security updates. An AI system can also fail quietly, after a model update, a change in source content, or a shift in what users ask. AI operations covers that work: rerunning evaluations, reviewing failures and feedback, validating changes before they become the default, and watching quality, latency, and cost together. Most production AI systems need both, each with a named owner.
Would anyone notice worse answers?
An AI system can keep running while its answers get worse. Uptime dashboards stay green. Requests return. What fails is answer quality, grounding, tool choice, permission behavior, latency, or cost.
Name the receiving owner first. Someone must be responsible for noticing whether the system still does the business task well. That owner decides whether an answer is good enough for that task and whether a change should pause the workflow. Without that owner, quiet failure becomes visible only after users lose trust.
Quiet failure can follow a model update, changed source content, new kinds of questions, or a permission that widened as connected sources grew. Managed support sees loud failures. AI operations sees the answers.
What managed support covers
Managed support covers what any live system needs: health review, alerts, triage, escalation, maintenance, dependency updates, security fixes, performance, capacity, agreed coverage, and explicit service levels.
It is the right shape for outages, errors, slow requests, failed releases, and operational care. It should name covered systems, coverage hours, response expectations, escalation contacts, and reporting.
For the defect versus operations boundary, compare managed support and a project warranty. A warranty question is different from operating a live system.
Managed support can still be essential for an AI system. It keeps the application, infrastructure, releases, and security work covered while AI operations checks whether the system remains useful.
The two scopes should meet during reviews. A technical incident can explain a quality drop, and a quality review can expose a technical issue before users escalate it through support channels.
What AI operations adds
AI operations adds checks for behavior that ordinary monitoring may not catch. Sometimes called LLMOps, it reruns the evaluation set on an agreed rhythm and before any model, source content, prompt, or workflow change becomes the default.
It reviews flagged answers, failures, and user feedback. It watches for provider model updates and reruns the checks when one lands. It reviews permissions as sources and tools grow.
It also watches cost and response time per task. A system can keep answering and still become too slow or too expensive for the work it serves.
The pause and fallback need to keep working. If automation should stop after a bad evaluation, name who can pause it and how. If the fallback is manual review, name the person or team that receives the work.
Treat it as recurring operating work. The same system may need review again when content changes, when users ask different questions, or when permissions expand.
UCS managed support for AI systems can include rerunning the agreed evaluation set after model, content, or workflow changes, and reviewing quality, latency, and cost.
Under UCS managed support, the operating scope can name which of these lines UCS covers, such as monitoring and evaluation reruns, and which stay with your team.
Compare by what fails and who notices
Use the comparison to name both responsibilities.
What fails
Managed support: Outages, errors, slow requests, failed releases
AI operations: Answer quality, grounding, tool choices, and cost
How you notice
Managed support: Alerts and monitoring
AI operations: Evaluation reruns, sampled reviews, and user feedback
Routine work
Managed support: Updates, security fixes, capacity reviews
AI operations: Change validation, failure reviews, permission reviews
What triggers action
Managed support: An alert or an incident
AI operations: A model, content, prompt, or workflow change, or a drop in evaluation results
Who judges
Managed support: Engineers against agreed service levels
AI operations: A subject expert against the agreed evaluation set
Most production AI systems need both. Name an owner for each column
Write service levels you can check
Classic service levels name covered systems, coverage hours, response expectations, escalation, and incident priorities. They should be explicit. Do not assume out-of-hours coverage, response times, or maintenance responsibilities.
AI service levels need different checks. Tie quality to the agreed evaluation set. Say when it is rerun, what triggers a pause or rollback, and who judges the results.
Cost needs an owner too. The question is not only whether the system is alive, but whether the response time and cost still make sense for the task.
Avoid vague promises. A service level should say what will be observed, who reviews it, and what action follows when it fails.
Service levels also need a path for disagreement. If the subject expert and technical owner read the evidence differently, the operating scope should say who decides whether to pause, adjust, or continue.
Who should own each part
Your side keeps the business outcome. A subject expert should judge whether answers are good enough for the task. Engineers can monitor systems, but they should not be the only judges of domain correctness.
Operations can sit with your team, a provider, or be split. Each task still needs a named person: who investigates failures, who changes permissions, who validates updates, who reviews feedback, and who can pause automation.
At UCS, managed support can cover agreed responsibilities for an AI system after launch. The operating scope names who investigates failures, who validates updates, and when a model, document, or workflow change needs another evaluation.
For the release handover, see what an AI system handover must include. Do not rely on a support agreement to create missing ownership after launch.
UCS can take over operations of an AI system another team built, after a health review that includes its evaluation set and permissions.
If responsibilities are split, make the handoff visible. A support engineer may see an alert, while the subject expert judges answer quality. The operating routine should connect those views.
A decision checklist
Who would notice if answer quality dropped?
Name the person who judges the answers, not only the person who sees alerts. If answer quality matters to a business task, a subject expert needs a regular way to review it and decide whether action is needed. That person receives the evidence.
When was the evaluation set last run, and by whom?
If nobody knows, the system may be operating without an AI-specific check. The evaluation set should be rerun before important changes become default and on an agreed rhythm owned by a named person. Record the result where the team can find it.
What happens when the model provider updates the model?
Decide who watches for the change, reruns the evaluation set, and approves continued use. A provider update is not automatically a defect in your code, but it can still change the system’s behavior. Treat the check as operating work, not research.
Who can pause automation, and how?
The pause path should be usable under pressure. Name the person who can pause, the condition that triggers it, and what users or operators do while automation is paused. If pausing needs approval, name the approver and the fallback path.
Who reviews cost and response time against what the task is worth?
Cost and latency should be judged against the job, not in isolation. Name who decides whether the system still earns its operating cost and whether slow answers change the user workflow. That judgment may sit with the business owner, not only engineering.
When another option fits better
Warranty: when the issue is a defect in the delivered scope, under the original agreement.
New scope: when you need substantial changes or new features. Managed support can carry agreed care, but a new capability needs its own acceptance.
Fractional leadership: when the open question is evaluation approach or data boundaries. An AI lead can own those decisions.
Your own team: when it can run the evaluation set, judge results, change permissions, and pause automation. Keep the business owner named either way.
Neither: when the system is not important enough to justify ongoing operation. Retire or narrow it rather than pretending it has production care.
Bring the system and how you judge it
Bring what the system does, who uses it, how answers are judged today, the evaluation set if one exists, known failures, and coverage needs. Do not send credentials, logs with personal data, or production data dumps.
You can also email hello@ucs.tech with how you judge the system’s answers today.
Questions before we start
Does managed support cover AI systems?
It covers what any live system needs: monitoring, incidents, updates, and agreed improvements. AI-specific work, such as rerunning evaluations, reviewing failures and feedback, and validating model or content changes, is covered only if the scope names it. Don’t assume it.
What is AI drift?
A change in an AI system’s behavior without a change in your code. It can follow a provider model update, new or edited source content, or a shift in what users ask. Catch it by rerunning an agreed evaluation set and reviewing real answers.
How often should we rerun AI evaluations?
Before any model, source content, prompt, or workflow change becomes the production default, and on a regular schedule agreed with the system’s owner. Also rerun after a provider announces a model change.
The model our AI system runs on is being retired. What do we do, and who does it?
Treat it as planned AI operations work, not an emergency. Choose a replacement and rerun the agreed evaluation set on it before anything changes in production. Adjust prompts or settings where results drop, then switch over in stages while you watch quality, response time, and cost. Keep the current model ready as a rollback until it is retired. Name one owner for the switch, whether your team or your operations provider, and a subject expert who approves the results.
What should an AI service level include?
The usual coverage hours, response expectations, and escalation, plus quality thresholds tied to an agreed evaluation set, when that set is rerun, and what triggers a pause or rollback. Agree who judges results and who reviews cost.
What does it cost to keep an AI system running after launch?
Running an AI system after launch has several cost lines, and each needs an owner. Model usage is usually billed by the provider to the system owner. Monitoring usually sits with whoever runs support. Evaluation reruns need engineering time plus a subject expert to judge the results. Model and prompt changes, including a move off a retired model, need engineering work and another evaluation before they become the default. Human review of flagged answers usually sits with your subject experts. Ask for each line by name when you scope support.
Should we run AI operations ourselves or use a provider?
Run it yourselves if your team can rerun the evaluation set, judge the results, change permissions, and pause automation. Use a provider for the parts your team can’t cover, and keep a named owner for the business outcome either way.
We inherited an AI system from a vendor or an engineer who left. What do we do first?
Start with a health review, not a rebuild. List what the system depends on: models, prompts, data sources, integrations, and the keys and accounts behind them. Move access, billing, and alerts to a named owner on your side, and remove access the previous builder no longer needs. Then rebuild an evaluation set from real cases, judged by a subject expert, so you can see how well the system works today. Discuss a rebuild only once that evidence exists.