If we move to 10, there may be not much point keeping the nicely scalable web instances/workers since we’ll also need to run kafka, zookeeper, etc. I was pondering something like:
Obviously, if the instance went down/had to be replaced we’d lose a bunch of stuff, but at least we wouldn’t lose our data. Or would we? Will Sentry be happy spinning up if it’s ‘lost’ kafka/zookeeper/redis … ? Would it help to keep the redis around as well?
You would lose data. Redis keeps many in-flight data such as pending jobs/tasks, kafka and zookeeper may also be holding in-flight event data. I also don’t see you mentioning Snuba and Clickhouse. All the event data is now stored in Clickhouse so you’d need to ensure its data is also safe.
Thanks! I should have clarified that we don’t care too much about losing in-flight data, we’d just like to not lose historical data. So if we had a resilient clickhouse and RDS, we could get away with something sketchy for the rest and not lose historical data? Would Sentry by ‘happy’ spinning up with an empty zookeeper/kafka/redis/snuba/whatever?
(sorry, I’m not familiar with Sentry 10’s architecture - I was just looking at the ‘whoah, there’s a lot of stuff in docker-compose’)
Just to make sure you understand the ramifications: you may lose very recent event data that is not fully committed to the datastore if you lose data from Kafka. If that’s fine, yeah with a resilient Postgres and Clickhouse datastore, you should be good.
Yeah it would as these are assumed to be ephemeral. All data required for a smooth run is stored in Postgres.