I know nothing about this helm chart, but tell them in any distributed system, it must have some central file storage. Whether you wanna NFS it yourself, or just offload to S3 or GCS (which is what I’d prefer).
Then you can worry less about backups. On top of that, if you care about your other state full datasets, I would not run them in Kubernetes, just so you have more control over backup strategies. Like, I would not run Postgres, keep it on a VM or use CloudSQL and manage backup options there.
I guess my point is, in Sentry’s case, we generally find a generic “backup script” to be difficult since everyone’s situation is difficult. We also potentially have terabytes of data spread across multiple systems, and each system probably has a better backup strategy. For example, if you’re running Postgres on a VM, disk snapshotting is great and much simpler than a SQL dump or something like that. Same with running systems like Kafka or Clickhouse in the future. It’s just extremely nontrivial to backup the world in a single command, and probably impractical.
I hope this adds some more clarifications. If you need any help with configuration of filestore options, let me know. I don’t think we document them too well, and honestly, maybe not at all for GCS.