Traditional software mostly behaves the same tomorrow as it does today. AI systems do not. The world changes, the data changes, models are updated and people find new ways to use the system. Launch is where the learning starts.
What changes after launch
The data moves. Customer behaviour, products, regulations and documents all evolve. A system tuned on last year’s reality can quietly become less accurate.
The usage surprises you. Real users ask things the test set never did. Some of those cases matter a great deal.
The costs shift. Token usage, retrieval volume and infrastructure scale with adoption, sometimes faster than expected.
The components change. Model providers release new versions and retire old ones. Each change can alter behaviour.
What to put in place
- Evaluation. A set of realistic test cases, run before every change and on a regular schedule, so quality is measured and not assumed.
- Monitoring. Track quality indicators, latency, cost and errors, and keep traces so you can see what the system did and why.
- Feedback loops. Make it easy for users to flag bad answers, and review them.
- Guardrails. Controls that keep the system within its purpose and filter unsafe or sensitive content.
- Ownership. A named person accountable for quality, cost and risk, with the authority to pause or roll back.
The AWS Well-Architected Framework applies here too: operational excellence, security, reliability, performance efficiency and cost optimisation all matter for AI workloads.
How we approach it
We treat operations as part of the design, not a phase after it. Our Managed Services and Forward Deployed Engineering work keep the same engineers accountable through go-live and afterwards.