r/mlops
[D] What's actually in your model rollback runbook?
- upvotes
- 6
- comments
- 4
Post
Everyone has a deploy process. I'm curious how many teams have a rollback that someone on call at 2am can run without the person who trained the model. What do you roll back to: a registry version, an image tag, a digest? Does the data/feature version roll back with it, or just the weights? How long does it take in practice? Ever had a rollback fail because the "old" version was overwritten or garbage collected? For context on my bias: I maintain KitOps, where the model, dataset reference, code, and config get packed into one versioned OCI artifact (a ModelKit), so "roll back" means pointing at the previous digest. That covers the artifact, but it doesn't solve feature stores or schema changes, and I'd like to know how people handle those parts. Stories about the time it didn't work are welcome.
Extracted from these lines
[comment u/Academic-Sun9444] I use Baseten to deploy. It allows to keep previous deployments without running them.
[comment u/Academic-Sun9444] If anything goes wrong we can automate to rollback to previous version or do it manually via the UI. It takes about 10-20 minutes depending on the model.