Unable to capture event in Sentry 20.6.0

I am trying to install the sentry using ./install.sh.
After the install I am able to open the web page and create the project, than I try to use the client DNS key to send events to sentry from my C# project, but it is not doing anythings.
(I tried the “Create a sample event” which is working fine)

Then I started to check the containers
All the containers are up except for the below four

  • sentry_onpremise_sentry-cleanup_1
  • sentry_onpremise_snuba-cleanup_1
  • sentry_onpremise_symbolicator-cleanup_1
  • sentry_onpremise_relay_1

For those cleanup I assume they only run for a short while then it will restart again.
For relay, I am not sure it is normal behavior.

The only configration I changed is the stmp server, which is working fine, I try to invite new members, it did sent out the email.

Then I try to check the logs in each container
All the snuba containers complains about the connection to kafka fail. Not sure it is related to the missing events.

below is the log
2020-08-20 03:51:17,003 New partitions assigned: {Partition(topic=Topic(name='event-replacements'), index=0): 0}

2020-08-20 03:51:23,910 Partitions revoked: [Partition(topic=Topic(name='event-replacements'), index=0)]

+ '[' r = - ']'

+ snuba replacer --help

+ set -- snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3

+ set gosu snuba snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3

+ exec gosu snuba snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3

%3|1597895567.301|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#${myip}:9092 failed: Connection refused (after 0ms in state CONNECT)

%3|1597895568.299|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#${myip}:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)

2020-08-20 03:53:00,310 New partitions assigned: {Partition(topic=Topic(name='event-replacements'), index=0): 0}

I also tried install the newer version from clean install, the same issues happens

  • 20.7.2
  • 20.8.0

They all should be up and running at all times. Each cleanup service has its own cron running that triggers the actual clean up jobs periodically. relay is the core event ingestion service so if that’s down, you won’t receive events at all.

It 100% is. Seems like your kafka service is having issues staying up. It may be a disk or a memory limitation. Once you manage to keep kafka up and running, simply restart other services and things should be fine.

I manage to run the relay container, that is due to the mssing credentials.json file.

Now I am able to get the events.

But the kafka connection issue still the same, the connection fails is happening when running the install.sh script. I think I got enough ram for that box (the suggested ram is 2.4gb I got 4gb ram)

It fails when it tries to Bootstrapping and migrating Snuba…
I am not able to attach the log file here, so I posted part of the failure part if that helps (I changed my ip to ${Myip})

BTW What would be the result of the disconnection from Kafka, I am still able to get the events now, does it mean are the events in there will stay forever, as the clean up job sounds like removing some old stuff.

Thanks for your help.

Docker images built.
Removing network onpremise_default
Network onpremise_default not found.
Removing network sentry_onpremise_default
Creating network “sentry_onpremise_default” with the default driver
Creating sentry_onpremise_clickhouse_1 …
Creating sentry_onpremise_clickhouse_1 … done
Trying again. Remaining tries #4
Bootstrapping and migrating Snuba…
Starting sentry_onpremise_clickhouse_1 …
Creating sentry_onpremise_redis_1 …
Starting sentry_onpremise_clickhouse_1 … done
Creating sentry_onpremise_zookeeper_1 …
Creating sentry_onpremise_zookeeper_1 … done
Creating sentry_onpremise_kafka_1 …
Creating sentry_onpremise_redis_1 … done
Creating sentry_onpremise_kafka_1 … done

  • ‘[’ b = - ‘]’
  • snuba bootstrap --help
  • set – snuba bootstrap --force
  • set gosu snuba snuba bootstrap --force
  • exec gosu snuba snuba bootstrap --force
    %3|1597973567.254|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#{MyIp}:9092 failed: Connection refused (after 1ms in state CONNECT) %3|1597973568.253|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#{MyIp}:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
    2020-08-21 01:32:48,253 Connection to Kafka failed (attempt 0)
    Traceback (most recent call last):
    File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 58, in bootstrap
    client.list_topics(timeout=1)
    cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str=“Failed to get metadata: Local: Broker transport failure”}
    %3|1597973569.256|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#{MyIp}:9092 failed: Connection refused (after 1ms in state CONNECT) 2020-08-21 01:32:50,257 Connection to Kafka failed (attempt 1) Traceback (most recent call last): File "/usr/src/snuba/snuba/cli/bootstrap.py", line 58, in bootstrap client.list_topics(timeout=1) cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"} %3|1597973570.262|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#{MyIp}:9092 failed: Connection refused (after 4ms in state CONNECT, 1 identical error(s) suppressed)
    %3|1597973571.274|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#{MyIp}:9092 failed: Connection refused (after 11ms in state CONNECT) %3|1597973572.263|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#{MyIp}:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
    2020-08-21 01:32:52,263 Connection to Kafka failed (attempt 2)
    Traceback (most recent call last):
    File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 58, in bootstrap
    client.list_topics(timeout=1)
    cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str=“Failed to get metadata: Local: Broker transport failure”}
    %3|1597973573.275|FAIL|rdkafka#producer-4| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#{MyIp}:9092 failed: Connection refused (after 5ms in state CONNECT) 2020-08-21 01:32:54,270 Connection to Kafka failed (attempt 3) Traceback (most recent call last): File "/usr/src/snuba/snuba/cli/bootstrap.py", line 58, in bootstrap client.list_topics(timeout=1) cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"} %3|1597973574.271|FAIL|rdkafka#producer-4| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#{MyIp}:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
    %3|1597973575.274|FAIL|rdkafka#producer-5| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#${MyIp}:9092 failed: Connection refused (after 1ms in state CONNECT)
    2020-08-21 01:32:56,276 Connection to Kafka failed (attempt 4)
    Traceback (most recent call last):
    File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 58, in bootstrap
    client.list_topics(timeout=1)
    cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str=“Failed to get metadata: Local: Broker transport failure”}
    %5|1597973577.276|REQTMOUT|rdkafka#producer-5| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Timed out ApiVersionRequest in flight (after 1002ms, timeout #0)
    %4|1597973577.276|FAIL|rdkafka#producer-5| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: ApiVersionRequest failed: Local: Timed out: probably due to broker version < 0.10 (see api.version.request configuration) (after 1002ms in state APIVERSION_QUERY)
    %4|1597973577.276|REQTMOUT|rdkafka#producer-5| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
    2020-08-21 01:32:58,221 Failed to create topic cdc
    Traceback (most recent call last):
    File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 94, in bootstrap
    future.result()
    File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 439, in result
    return self.__get_result()
    File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 388, in __get_result
    raise self._exception
    cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str=“Topic ‘cdc’ already exists.”}

The install script should be taking care of this, doesn’t it:

These are transient errors when waiting for Kafka to be up and running. If you look closer, the errors are actually ignored on purpose and they switch to things like TOPIC_ALREADY_EXISTS etc. indicating that Kafka is reachable.

If Kafka was down, you wouldn’t be able to receive any events so looks like you don’t have any issues at the moment :slight_smile:

My pleasure :slight_smile:

I have exactly the same problems. Could you tell us how you solved them?
Thanks in advance!

The first issue I have is the credentials.json got miss configrated at the first place.

For the Kafka issue, I am not entirly sure but after fixing the first issues, this nolonger happens.

That could also happens if the hardware is not good enough (2.4 gb ram)