MLOps: Automating the Entire Training Loop

How I built a pipeline where a git push triggers retraining, validation, and redeployment on Kubernetes — zero manual steps, ever.


The problem with manual retraining

The standard ML workflow in most student and small-team projects looks something like this: train a model locally, evaluate it manually, copy the weights somewhere, update a config, and redeploy by hand. It works for a first version. It falls apart the moment you want to iterate.

The goal of this project was to build something where none of that is manual. Every code push — whether it changes the model architecture, the training script, or the data preprocessing — should trigger a full retrain, run validation automatically, and redeploy if the new model meets the quality bar. No human in the loop between push and production.

The architecture

The full pipeline runs across three main components:

Triggering retraining from a git push

The GitHub Actions workflow is the backbone of the automation. On push to main, it runs a sequence of steps: install dependencies, authenticate to Azure, submit the training job, poll for completion, pull the output model, run validation, and conditionally deploy.

The conditional is important. Automatic redeployment only happens if the new model passes a set of validation checks — minimum accuracy on a held-out test set, no regression on known failure cases, and a sanity check that the model artefact is the correct format. If any check fails, the workflow exits with a non-zero status and the existing production model stays up. The developer gets a notification and investigates.

Validation before deployment

Automated validation is the part that makes fully-automatic redeployment safe enough to ship. Without it, a broken training run would go straight to production.

The checks I implemented were deliberately simple but effective: threshold accuracy on a fixed test split, a regression suite of images the previous model handled correctly (to catch regressions in data loading or preprocessing), and a schema check on the output artefact. This isn't exhaustive, but it catches the majority of failure modes that actually occur: bugs in data loading, hyperparameter mistakes that tanked accuracy, and corrupted output files.

Redeployment on Kubernetes

The production environment runs FastAPI behind NGINX on a Kubernetes cluster. The deployment step in the workflow updates a Kubernetes deployment to use the new model. In practice this means updating a ConfigMap or volume mount pointing to the new weights, then triggering a rolling restart of the FastAPI pods.

Rolling restarts mean zero downtime — Kubernetes spins up new pods with the updated model before terminating the old ones. Requests continue to be served throughout the update. This was straightforward to configure but easy to get wrong: the readiness probe on the FastAPI pod has to correctly signal when the model is loaded and ready to serve, otherwise Kubernetes will route traffic to pods that aren't ready yet.

What this changes

The practical impact of a pipeline like this is hard to overstate once you've worked with one. Iteration speed goes up dramatically. You can try a different learning rate, push, and have a validated result in production (or a notification that it failed) without touching a terminal beyond the commit. More importantly, the pipeline forces you to define what "good enough to deploy" means — the validation step makes that explicit and enforces it automatically.

The same architecture scales. Adding a new dataset, a new model variant, or a new deployment region is a matter of extending the workflow, not rebuilding it from scratch.

What I'd do differently

The main limitation of this pipeline is the cost of compute for every push. In a cost-conscious environment, you'd want to gate the retraining trigger more carefully — perhaps only on changes to the model or data directories, not every file in the repo. GitHub Actions path filters make this straightforward to add.

I'd also add experiment tracking from the start. Azure ML has native support for tracking metrics across runs; I integrated it late in the project and retrofitting it into an existing training script is messier than designing for it upfront.