The problem with manual retraining
The standard ML workflow in most student and small-team projects looks something like this: train a model locally, evaluate it manually, copy the weights somewhere, update a config, and redeploy by hand. It works for a first version. It falls apart the moment you want to iterate.
The goal of this project was to build something where none of that is manual. Every code push — whether it changes the model architecture, the training script, or the data preprocessing — should trigger a full retrain, run validation automatically, and redeploy if the new model meets the quality bar. No human in the loop between push and production.
The architecture
The full pipeline runs across three main components:
- Azure ML — handles compute for training jobs. When a retrain is triggered, a training job is submitted to Azure ML which spins up the required compute, runs the training script, and stores the resulting model artefact.
- GitHub Actions — orchestrates the pipeline. A push to the main branch triggers a workflow that submits the Azure ML job, waits for completion, runs validation checks on the output model, and — if validation passes — kicks off redeployment.
- Kubernetes (via FastAPI + NGINX) — serves the model in production. The deployment step updates the running pod with the new model weights. NGINX handles routing and load balancing in front of the FastAPI application.
Triggering retraining from a git push
The GitHub Actions workflow is the backbone of the automation. On push to main, it runs a sequence of steps: install dependencies, authenticate to Azure, submit the training job, poll for completion, pull the output model, run validation, and conditionally deploy.
The conditional is important. Automatic redeployment only happens if the new model passes a set of validation checks — minimum accuracy on a held-out test set, no regression on known failure cases, and a sanity check that the model artefact is the correct format. If any check fails, the workflow exits with a non-zero status and the existing production model stays up. The developer gets a notification and investigates.
Validation before deployment
Automated validation is the part that makes fully-automatic redeployment safe enough to ship. Without it, a broken training run would go straight to production.
The checks I implemented were deliberately simple but effective: threshold accuracy on a fixed test split, a regression suite of images the previous model handled correctly (to catch regressions in data loading or preprocessing), and a schema check on the output artefact. This isn't exhaustive, but it catches the majority of failure modes that actually occur: bugs in data loading, hyperparameter mistakes that tanked accuracy, and corrupted output files.
Redeployment on Kubernetes
The production environment runs FastAPI behind NGINX on a Kubernetes cluster. The deployment step in the workflow updates a Kubernetes deployment to use the new model. In practice this means updating a ConfigMap or volume mount pointing to the new weights, then triggering a rolling restart of the FastAPI pods.
Rolling restarts mean zero downtime — Kubernetes spins up new pods with the updated model before terminating the old ones. Requests continue to be served throughout the update. This was straightforward to configure but easy to get wrong: the readiness probe on the FastAPI pod has to correctly signal when the model is loaded and ready to serve, otherwise Kubernetes will route traffic to pods that aren't ready yet.
What this changes
The practical impact of a pipeline like this is hard to overstate once you've worked with one. Iteration speed goes up dramatically. You can try a different learning rate, push, and have a validated result in production (or a notification that it failed) without touching a terminal beyond the commit. More importantly, the pipeline forces you to define what "good enough to deploy" means — the validation step makes that explicit and enforces it automatically.
The same architecture scales. Adding a new dataset, a new model variant, or a new deployment region is a matter of extending the workflow, not rebuilding it from scratch.
What I'd do differently
The main limitation of this pipeline is the cost of compute for every push. In a cost-conscious environment, you'd want to gate the retraining trigger more carefully — perhaps only on changes to the model or data directories, not every file in the repo. GitHub Actions path filters make this straightforward to add.
I'd also add experiment tracking from the start. Azure ML has native support for tracking metrics across runs; I integrated it late in the project and retrofitting it into an existing training script is messier than designing for it upfront.