Sentry no more catch errors

On-Premise
10

In some cases, the Post Process Forwarder gets into a bad state and you need to do some manual cleanup.

From Post Process Forwarder - KafkaError "Offset Out of Range" · Issue #478 · getsentry/self-hosted · GitHub

(This is if you haven’t changed the default number of instances from the main docker-compose, and if you are in the onpremise directory)

  1. Stop sentry

    docker-compose stop

  2. Start only Zookeeper and Kafka

    docker start sentry_onpremise_kafka_1 sentry_onpremise_zookeeper_1

  3. Hop into the Kafka instance interactively

    docker exec -it sentry_onpremise_kafka_1 /bin/bash

  4. Receive consumers list

    kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --list

  5. Get group info

    kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --group snuba-post-processor -describe

  6. Set the offsets to latest

    kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --group snuba-post-processor --topic events --reset-offsets --to-latest --execute

  7. Exit the Kafka instance

    exit

  8. Stop Zookeeper and Kafka

    docker stop sentry_onpremise_kafka_1 sentry_onpremise_zookeeper_1

  9. Restart the entire stack

    docker-compose up

  10. Check the logs. You should see the standard retry errors as it starts up, but the continuous restarts should be resolved.
4 Likes