The best way to upgrade to latest Sentry version

Am2020 · December 28, 2020, 6:19pm

Hi,
I have Sentry in Docker(Ec2) (sentry:9.0-onbuild), the Sentry points to external RDS Postgres. I want to upgrade to sentry’s latest version, what’s the recommendation to upgrade from Sentry 9 to latest: shall I create clean installation then restore DB/data? or upgrade Sentry 9 to latest directly

BYK · January 4, 2021, 10:46am

I’d try upgrading directly and running migrations at each step. You should be able to do 9.0.0 -> 9.1.2 -> 20.12.1

Am2020 · January 4, 2021, 11:05am

Hi,
I tried to do that to test the upgrade of Sentry 20:

upgrade from 9.0.0 to 9.1.2
1.a backup the current Postgres DB and Sentry-data volume)
1.b get the latest version of this repo.
1.c use the following steps after updating this repository:
docker-compose build --pull
docker-compose run --rm web upgrade
docker-compose up -d
upgrade from 9.1.2 to 10
2.a backup the Postgres DB and Sentry-data volume)
2.b get the latest version of this repo.
2.c run ./install.sh script
upgrade from 10 to 20
3.a backup the Postgres DB and Sentry-data volume)
3.b get the latest version of this repo.
3.c run ./install.sh script

I noticed that during the upgrade (using the above steps)

to upgrade from 9.1.2 to 10, the server run out of memory (the memory was 30G — >> 16G) why the upgrade consume this memory
found this error during upgrading to Sentry (10)

File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 50, in bootstrap^M

    client.list_topics(timeout=1)^M
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}^M
2020-12-30 22:04:19,761 Connection to Kafka failed (attempt 1)^M
Traceback (most recent call last):^M
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 50, in bootstrap^M
    client.list_topics(timeout=1)^M
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}^M

Also this error during upgrading to sentry 20

2020-12-30 23:22:55,902 Connection to Kafka failed (attempt 1)^M

Traceback (most recent call last):^M
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap^M
    client.list_topics(timeout=1)^M
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}^M
2020-12-30 23:22:57,147 Failed to create topic cdc^M
Traceback (most recent call last):^M
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap^M
    future.result()^M
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 439, in result^M
    return self.__get_result()^M
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result^M
    raise self._exception^M
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'cdc' already exists."}^M
2020-12-30 23:22:57,148 Failed to create topic events^M
Traceback (most recent call last):^M
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap

Also the relay/geoip containers aren’t in up
Relay log container:

error: could not parse json config file (file /work/.relay/credentials.json)
caused by: expected value at line 1 column 1

GeoIP log container

error loading configuration file /sentry/GeoIP.conf: error opening file: open /sentry/GeoIP.conf: no such file or directory

could I upgrade directly from 9.1.2 to 20? because when I tried that, the ./install script showed the Kafka error in loop then failed

Am2020 · January 4, 2021, 11:15am

@BYK u can check the logs from this link: https://we.tl/t-3RPXhUg9mE
BTW I tried the below steps to solve the Infinite Kafka loop while upgrading from 9.1.2 to 20 but that didn’t help.

docker-compose down
docker volume rm sentry-kafka sentry-zookeeper
docker volume rm sentry_onpremise_sentry-kafka-log sentry_onpremise_sentry-zookeeper-log
./install.sh

BYK · January 4, 2021, 7:39pm

Yes, that’s why I didn’t mention v10. I’d actually avoid going through v10 as we have refined the upgrade path in versions later than 10.

If you have too many events, then running out of memory is understandable. You may run the install script with SENTRY_SKIP_EVENTS_BACKFILL_FOR_10=1 to skip migration of old events. Note that they will all get deleted after the upgrade, so beware.

Am2020 · January 4, 2021, 8:04pm

@BYK but when i upgraded from 9.1.2 to 20, i got the Infinite Kafka loop error then the installation script failed

cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str=“Failed to get metadata: Local: Broker transport failure”}^M
%3|1609326160.925|FAIL|rdkafka#producer-57| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve ‘kafka:9092’: No address associated with hostname (after 3ms in state CONNECT, 1 identical error(s) suppressed)^M
%3|1609326160.926|FAIL|rdkafka#producer-58| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve ‘kafka:9092’: No address associated with hostname (after 2ms in state CONNECT)^M
2020-12-30 11:02:41,925 Connection to Kafka failed (attempt 57)^M
Traceback (most recent call last):^M
File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 55, in bootstrap^M
client.list_topics(timeout=1)^M
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str=“Failed to get metadata: Local: Broker transport failure”}^M
%3|1609326162.927|FAIL|rdkafka#producer-58| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve ‘kafka:9092’: No address associated with hostname (after 3ms in state CONNECT, 1 identical error(s) suppressed)^M
%3|1609326162.929|FAIL|rdkafka#producer-59| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve ‘kafka:9092’: No address associated with hostname (after 2ms in state CONNECT)^M
2020-12-30 11:02:43,928 Connection to Kafka failed (attempt 58)^M
Traceback (most recent call last):^M
File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 55, in bootstrap^M
client.list_topics(timeout=1)^M
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str=“Failed to get metadata: Local: Broker transport failure”}^M
%3|1609326164.930|FAIL|rdkafka#producer-59| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve ‘kafka:9092’: No address associated with hostname (after 3ms in state CONNECT, 1 identical error(s) suppressed)^M
%3|1609326164.932|FAIL|rdkafka#producer-60| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve ‘kafka:9092’: No address associated with hostname (after 3ms in state CONNECT)^M
2020-12-30 11:02:45,931 Connection to Kafka failed (attempt 59)^M
Traceback (most recent call last):^M
File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 55, in bootstrap^M
client.list_topics(timeout=1)^M
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str=“Failed to get metadata: Local: Broker transport failure”}^M
Traceback (most recent call last):^M
File “/usr/local/bin/snuba”, line 33, in ^M
sys.exit(load_entry_point(‘snuba’, ‘console_scripts’, ‘snuba’)())^M
File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 722, in call^M
return self.main(*args, **kwargs)^M
File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 697, in main^M
rv = self.invoke(ctx)^M
File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 1066, in invoke^M
return _process_result(sub_ctx.command.invoke(sub_ctx))^M
File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 895, in invoke^M
return ctx.invoke(self.callback, **ctx.params)^M
File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 535, in invoke^M
return callback(*args, **kwargs)^M
File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 55, in bootstrap^M
client.list_topics(timeout=1)^M
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str=“Failed to get metadata: Local: Broker transport failure”}^M
An error occurred, caught SIGERR on line 251
Cleaning up…

i tried to stop docker and delete volumes but that didn’t help

docker-compose down
docker volume rm sentry-kafka sentry-zookeeper
docker volume rm sentry_onpremise_sentry-kafka-log sentry_onpremise_sentry-zookeeper-log
./install.sh

BYK · January 5, 2021, 11:55am

Could it be ZooKeeper that is failing?

Am2020 · January 5, 2021, 10:27pm

@BYK
I tried to re-upgrade 9.1.2 to 20 and this time i didn’t see the Kafka error but i saw the below errors:
1.

exec gosu snuba snuba bootstrap --no-migrate --force^M

%3|1609881376.535|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#10.201.6.5:9092 failed: Connection refused (after 1ms in state CONNECT)^M

%3|1609881377.534|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#10.201.6.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)^M

2021-01-05 21:16:17,534 Connection to Kafka failed (attempt 0)^M

Traceback (most recent call last):^M

File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 55, in bootstrap^M

client.list_topics(timeout=1)^M

cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str=“Failed to get metadata: Local: Broker transport failure”}^M

%3|1609881378.536|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#10.201.6.5:9092 failed: Connection refused (after 0ms in state CONNECT)^M

%3|1609881379.536|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#10.201.6.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)^M

2021-01-05 21:16:19,537 Connection to Kafka failed (attempt 1)^M

Traceback (most recent call last):^M

/usr/local/lib/python3.6/site-packages/sentry/runner/initializer.py:189: DeprecatedSettingWarning: The EMAIL_USE_SSL setting is deprecated. Please use SENTRY_OPTIONS[‘mail.use-ssl’] instead.^M
warnings.warn(DeprecatedSettingWarning(options_mapper[k], “SENTRY_OPTIONS[‘%s’]” % k))^M
/usr/local/lib/python3.6/site-packages/sentry/runner/initializer.py:189: DeprecatedSettingWarning: The GITHUB_EXTENDED_PERMISSIONS setting is deprecated. Please use SENTRY_OPTIONS[‘github-login.extended-permissions’] instead.^M

sentry_onpremise_relay_1 container in restarting status
82119149b902 getsentry/relay:20.12.1 “/bin/bash /docker-e…” 52 minutes ago Restarting (1) 33 seconds ago sentry_onpremise_relay_1

error: could not parse json config file (file /work/.relay/credentials.json)
caused by: expected value at line 1 column 1
error: could not parse json config file (file /work/.relay/credentials.json)
caused by: expected value at line 1 column 1

sentry_onpremise_geoipupdate_1 container in Exited status
211904d3ec02 maxmindinc/geoipupdate:latest “/usr/bin/geoipupdat…” 54 minutes ago Exited (1) 54 minutes ago sentry_onpremise_geoipupdate_1

error loading configuration file /sentry/GeoIP.conf: error opening file: open /sentry/GeoIP.conf: no such file or directory

could u advise why I got these errors?
Also I have some question:

which permission that I should set to repo files before run ./install.sh? 777 is not the best practices
if I have loadblancer should, can i remove Nginx container from docker-compose.yml ?
does the Sentry 20 save the data in DB like Sentry 9 or it’s save in docker volumes?

BYK · January 7, 2021, 7:31pm

Errors from this one should be safe to ignore as they are transient errors while waiting for Kafka to be up and running. These are printed for debugging purposes but I agree they are a common source of confusion.

This is a false warning that is safe to ignore. It will go away in the next version.

This is a legit warning. Just do what it says or move this setting to config.yml file. Pretty straightforward.

This means your relay/config.yml file is invalid (probably incorrectly formatted). I’d check its contents to see what is wrong. Also, unless you fix this issue you won’t be able to receive any events as Relay is the one that processes them.

This is okay to ignore unless you want to use GeoIP to resolve errors and performance traces to geolocations. See https://develop.sentry.dev/self-hosted/geolocation/ for more information.

As long as you can execute the .sh files, read all the files and write under the directories in that repo as your current user, it should be fine.

No, use it in front of the existing Nginx instance. It is there for routing and some HTTP processing, not load balancing.

It saves the data in many different places, some are databases like Postgres or ClickHouse and all that data lives in Docker Volumes for persistence.

Am2020 · January 14, 2021, 11:40am

Hi @BYK
what about this error

%3|1609881377.534|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#10.201.6.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)^M

This means your relay/config.yml file is invalid (probably incorrectly formatted). I’d check its contents to see what is wrong. Also, unless you fix this issue you won’t be able to receive any events as Relay is the one that processes them.

i removed the corrupted file and re-run the installation and still the relay container is not up

2021-01-14T15:12:57Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#10.201.3.9:9092 failed: Connection refused (after 2ms in state CONNECT)

2021-01-14T15:12:57Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#10.201.3.9:9092 failed: Connection refused (after 2ms in state CONNECT)

Am2020 · January 14, 2021, 11:08pm

@BYK just quick update:
the relay container is up now after configuring the Geoi as per this link and I can receive the events now but I see the sentry_onpremise_geoipupdate container is in Exist status, how can I solve this error

docker-compose logs -f geoipupdate

Attaching to sentry_onpremise_geoipupdate_1

sentry_onpremise_geoipupdate_1 exited with code 0

BYK · January 15, 2021, 11:41am

This looks like a @chadwhitacre question who implemented this feature so I’ll let him chime in

Am2020 · January 19, 2021, 8:26am

@BYK @chadwhitacre is it bug in Sentry(20) or i am missing something in Geoi configuration? and how does this error affect the Sentry functionality/performance?

chadwhitacre · January 19, 2021, 8:42pm

That’s expected behavior @Am2020, sorry for the confusion. The container runs once during installation/upgrade to download the latest data file from MaxMind, it is not a long-lived container (the long-lived web and relay containers make use of the updated data file).

chadwhitacre · January 19, 2021, 8:44pm

P.S. I added a note to Write a doc that explains all of the parts of Sentry · Issue #789 · getsentry/onpremise · GitHub to document container life-span there.

Am2020 · January 23, 2021, 12:00pm

@BYK @chadwhitacre
I have upgrade to Sentry(20) on prod and seems all works fine, I can send/receive events but I see this error in relay container log, is it okay or I have an issue in the Sentry stack?

2021-01-23T11:35:21Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: could not send request using actix-web client
caused by: Failed to connect to host: Connection refused (os error 111)
caused by: Connection refused (os error 111)
caused by: Connection refused (os error 111)
2021-01-23T11:35:29Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: could not send request using actix-web client
caused by: Server disconnected
caused by: Server disconnected

BYK · January 23, 2021, 8:42pm

This is fine as long is it doesn’t repeat indefinitely (so it is transient) as relay starts much faster than sentry-web.

system · February 7, 2021, 8:43pm

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Best way to upgrade to latest sentry version from v9.0.0	6	2331	June 1, 2021
Upgrading from 9.12 -> 10.0.0 -> v20.6.0 On-Premise	27	7933	October 24, 2020
Sentry 9.1.2 is out! On-Premise	8	6581	November 30, 2019
No events / issue details after upgrade from 9.1.2 to 20.9/latest On-Premise	16	4440	October 27, 2020
Upgrading from sentry 9.1.2 to latest sentry installation On-Premise	2	1276	May 9, 2021

The best way to upgrade to latest Sentry version

Related Topics