Model registry, drift monitoring and an automatic retraining trigger

Kestrel predicts hourly rental demand for each of its 420 docking zones, so night crews know where to move scooters. The model was trained once, eight months ago, by a data scientist who has since left. Nobody knows which data or code produced the file in production. Since a new city opened and a competitor arrived, the operations team says the predictions "feel off", but nobody can prove it, because no one compares predictions with actual rentals.

The head of data wants the model treated like software. Every training run is logged. Every production model can be traced to its data, code and metrics. Drift is checked daily. When performance drops past a set limit, retraining starts automatically, but the new model only replaces the old one if it is better on a recent holdout.

You will use the public UCI Bike Sharing dataset (hourly rentals with weather and calendar fields) as Kestrel's history, and replay it day by day to simulate production, including a deliberate shift you introduce, such as a new-city effect or a weather pattern change.

You will build: - experiment tracking and a model registry with MLflow; - a batch prediction job; - data-drift checks (PSI on key features) and performance monitoring (MAE against actuals with a 48-hour delay); - a retraining pipeline triggered when 7-day MAE rises more than 20% above the registered model's validation MAE; - a promotion gate and one-command rollback.

Done looks like a repo that runs the whole loop locally with Docker Compose, a replay over 120 simulated days showing at least one drift alert and one automatic retrain, and a runbook the on-call engineer can follow.