Unable to capture event in Sentry 20.6.0

WillLiu · August 20, 2020, 4:31am

I am trying to install the sentry using ./install.sh.
After the install I am able to open the web page and create the project, than I try to use the client DNS key to send events to sentry from my C# project, but it is not doing anythings.
(I tried the “Create a sample event” which is working fine)

Then I started to check the containers
All the containers are up except for the below four

sentry_onpremise_sentry-cleanup_1
sentry_onpremise_snuba-cleanup_1
sentry_onpremise_symbolicator-cleanup_1
sentry_onpremise_relay_1

For those cleanup I assume they only run for a short while then it will restart again.
For relay, I am not sure it is normal behavior.

The only configration I changed is the stmp server, which is working fine, I try to invite new members, it did sent out the email.

Then I try to check the logs in each container
All the snuba containers complains about the connection to kafka fail. Not sure it is related to the missing events.

below is the log
2020-08-20 03:51:17,003 New partitions assigned: {Partition(topic=Topic(name='event-replacements'), index=0): 0}

2020-08-20 03:51:23,910 Partitions revoked: [Partition(topic=Topic(name='event-replacements'), index=0)]

+ '[' r = - ']'

+ snuba replacer --help

+ set -- snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3

+ set gosu snuba snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3

+ exec gosu snuba snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3

%3|1597895567.301|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#${myip}:9092 failed: Connection refused (after 0ms in state CONNECT)

%3|1597895568.299|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#${myip}:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)

2020-08-20 03:53:00,310 New partitions assigned: {Partition(topic=Topic(name='event-replacements'), index=0): 0}

I also tried install the newer version from clean install, the same issues happens

20.7.2
20.8.0

BYK · August 20, 2020, 8:04pm

They all should be up and running at all times. Each cleanup service has its own cron running that triggers the actual clean up jobs periodically. relay is the core event ingestion service so if that’s down, you won’t receive events at all.

It 100% is. Seems like your kafka service is having issues staying up. It may be a disk or a memory limitation. Once you manage to keep kafka up and running, simply restart other services and things should be fine.

WillLiu · August 21, 2020, 1:52am

I manage to run the relay container, that is due to the mssing credentials.json file.

Now I am able to get the events.

But the kafka connection issue still the same, the connection fails is happening when running the install.sh script. I think I got enough ram for that box (the suggested ram is 2.4gb I got 4gb ram)

It fails when it tries to Bootstrapping and migrating Snuba…
I am not able to attach the log file here, so I posted part of the failure part if that helps (I changed my ip to ${Myip})

BTW What would be the result of the disconnection from Kafka, I am still able to get the events now, does it mean are the events in there will stay forever, as the clean up job sounds like removing some old stuff.

Thanks for your help.

Docker images built.
Removing network onpremise_default
Network onpremise_default not found.
Removing network sentry_onpremise_default
Creating network “sentry_onpremise_default” with the default driver
Creating sentry_onpremise_clickhouse_1 …
Creating sentry_onpremise_clickhouse_1 … done
Trying again. Remaining tries #4
Bootstrapping and migrating Snuba…
Starting sentry_onpremise_clickhouse_1 …
Creating sentry_onpremise_redis_1 …
Starting sentry_onpremise_clickhouse_1 … done
Creating sentry_onpremise_zookeeper_1 …
Creating sentry_onpremise_zookeeper_1 … done
Creating sentry_onpremise_kafka_1 …
Creating sentry_onpremise_redis_1 … done
Creating sentry_onpremise_kafka_1 … done

‘[’ b = - ‘]’
snuba bootstrap --help
set – snuba bootstrap --force
set gosu snuba snuba bootstrap --force
exec gosu snuba snuba bootstrap --force
%3|1597973567.254|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#{MyIp}:9092 failed: Connection refused (after 1ms in state CONNECT) %3|1597973568.253|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#{MyIp}:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2020-08-21 01:32:48,253 Connection to Kafka failed (attempt 0)
Traceback (most recent call last):
File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 58, in bootstrap
client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str=“Failed to get metadata: Local: Broker transport failure”}
%3|1597973569.256|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#{MyIp}:9092 failed: Connection refused (after 1ms in state CONNECT) 2020-08-21 01:32:50,257 Connection to Kafka failed (attempt 1) Traceback (most recent call last): File "/usr/src/snuba/snuba/cli/bootstrap.py", line 58, in bootstrap client.list_topics(timeout=1) cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"} %3|1597973570.262|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#{MyIp}:9092 failed: Connection refused (after 4ms in state CONNECT, 1 identical error(s) suppressed)
%3|1597973571.274|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#{MyIp}:9092 failed: Connection refused (after 11ms in state CONNECT) %3|1597973572.263|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#{MyIp}:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2020-08-21 01:32:52,263 Connection to Kafka failed (attempt 2)
Traceback (most recent call last):
File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 58, in bootstrap
client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str=“Failed to get metadata: Local: Broker transport failure”}
%3|1597973573.275|FAIL|rdkafka#producer-4| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#{MyIp}:9092 failed: Connection refused (after 5ms in state CONNECT) 2020-08-21 01:32:54,270 Connection to Kafka failed (attempt 3) Traceback (most recent call last): File "/usr/src/snuba/snuba/cli/bootstrap.py", line 58, in bootstrap client.list_topics(timeout=1) cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"} %3|1597973574.271|FAIL|rdkafka#producer-4| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#{MyIp}:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
%3|1597973575.274|FAIL|rdkafka#producer-5| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#${MyIp}:9092 failed: Connection refused (after 1ms in state CONNECT)
2020-08-21 01:32:56,276 Connection to Kafka failed (attempt 4)
Traceback (most recent call last):
File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 58, in bootstrap
client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str=“Failed to get metadata: Local: Broker transport failure”}
%5|1597973577.276|REQTMOUT|rdkafka#producer-5| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Timed out ApiVersionRequest in flight (after 1002ms, timeout #0)
%4|1597973577.276|FAIL|rdkafka#producer-5| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: ApiVersionRequest failed: Local: Timed out: probably due to broker version < 0.10 (see api.version.request configuration) (after 1002ms in state APIVERSION_QUERY)
%4|1597973577.276|REQTMOUT|rdkafka#producer-5| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
2020-08-21 01:32:58,221 Failed to create topic cdc
Traceback (most recent call last):
File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 94, in bootstrap
future.result()
File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 439, in result
return self.__get_result()
File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 388, in __get_result
raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str=“Topic ‘cdc’ already exists.”}

BYK · August 21, 2020, 8:28am

The install script should be taking care of this, doesn’t it:

github.com

getsentry/self-hosted/blob/4b8d4244772c22f23a72036a6a0d86ca55bd5831/install.sh#L283-L293


      
          if [ ! -f "$RELAY_CREDENTIALS_JSON" ]; then
            echo ""
            echo "Generating Relay credentials..."
          
            # We need the ugly hack below as `relay generate credentials` tries to read the config and the credentials
            # even with the `--stdout` and `--overwrite` flags and then errors out when the credentials file exists but
            # not valid JSON. We hit this case as we redirect output to the same config folder, creating an empty
            # credentials file before relay runs.
            $dcr --no-deps -v $(pwd)/$RELAY_CONFIG_YML:/tmp/config.yml relay --config /tmp credentials generate --stdout > "$RELAY_CREDENTIALS_JSON"
            echo "Relay credentials written to $RELAY_CREDENTIALS_JSON"
          fi

These are transient errors when waiting for Kafka to be up and running. If you look closer, the errors are actually ignored on purpose and they switch to things like TOPIC_ALREADY_EXISTS etc. indicating that Kafka is reachable.

If Kafka was down, you wouldn’t be able to receive any events so looks like you don’t have any issues at the moment

My pleasure

chritsian · September 4, 2020, 11:09am

I have exactly the same problems. Could you tell us how you solved them?
Thanks in advance!

WillLiu · September 18, 2020, 3:47am

The first issue I have is the credentials.json got miss configrated at the first place.

For the Kafka issue, I am not entirly sure but after fixing the first issues, this nolonger happens.

That could also happens if the hardware is not good enough (2.4 gb ram)

system · December 8, 2020, 1:26pm

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Sentry stops processing events after upgrade 10.0 => 20.8.0.dev0ba2aa70 On-Premise	52	11437	December 8, 2020
Sentry does not receive events (kafka error) On-Premise	5	5479	March 24, 2021
Events not displaying On-Premise	13	5177	January 22, 2021
Events have stopped appearing On-Premise	5	2956	January 3, 2021
Sentry stopped displaying events in UI On-Premise	6	2184	June 8, 2020

Unable to capture event in Sentry 20.6.0

Related topics