Error during upgrade from Sentry 9.1.2 to Sentry 10

On-Premise
1

Hello,

Tried to upgrade from 9.1.2 to 10.
I have the following containers:

[root@sentry ~]# docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                    NAMES
b59f2ce4d8d8        onpremise_web_1        "/entrypoint.sh run …"   2 seconds ago       Up 1 second         0.0.0.0:9000->9000/tcp   heuristic_wescoff
99361ad414a2        onpremise_cron         "/entrypoint.sh run …"   8 months ago        Up 6 minutes        9000/tcp                 onpremise_cron_1
b356a50bd308        onpremise_worker       "/entrypoint.sh run …"   8 months ago        Up 6 minutes        9000/tcp                 onpremise_worker_1
584989fe665e        postgres:9.5           "docker-entrypoint.s…"   8 months ago        Up 6 minutes        5432/tcp                 onpremise_postgres_1
a034845caf8c        memcached:1.5-alpine   "docker-entrypoint.s…"   8 months ago        Up 6 minutes        11211/tcp                onpremise_memcached_1
507a5542b8a0        tianon/exim4           "docker-entrypoint.s…"   8 months ago        Up 6 minutes        25/tcp                   onpremise_smtp_1
1bf25fbf3f68        redis:3.2-alpine       "docker-entrypoint.s…"   8 months ago        Up 6 minutes        6379/tcp                 onpremise_redis_1

I went to /home/onpremise (here is the old git project)
I updated with git pull
Then I tried running ./install.sh to upgrade
But it fails:

Checking minimum requirements...
Stopping wonderful_euclid      ... 
Stopping onpremise_cron_1      ... 
Stopping onpremise_worker_1    ... 
Stopping onpremise_postgres_1  ... 
Stopping onpremise_memcached_1 ... 
Stopping onpremise_smtp_1      ... 
Stopping onpremise_redis_1     ... 
Stopping onpremise_cron_1      ... done
Stopping wonderful_euclid      ... done
Stopping onpremise_worker_1    ... done
Stopping onpremise_memcached_1 ... done
Stopping onpremise_smtp_1      ... done
Stopping onpremise_postgres_1  ... done
Stopping onpremise_redis_1     ... done
Removing wonderful_euclid       ... 
Removing naughty_pascal         ... 
Removing vigorous_sanderson     ... 
Removing optimistic_kepler      ... 
Removing elastic_galileo        ... 
Removing inspiring_curie        ... 
Removing agitated_mclean        ... 
Removing optimistic_aryabhata   ... 
Removing adoring_chatelet       ... 
Removing lucid_noether          ... 
Removing cranky_volhard         ... 
Removing nervous_kapitsa        ... 
Removing eloquent_turing        ... 
Removing lucid_dubinsky         ... 
Removing crazy_brattain         ... 
Removing blissful_khorana       ... 
Removing elegant_blackburn      ... 
Removing flamboyant_babbage     ... 
Removing heuristic_varahamihira ... 
Removing exciting_burnell5      ... 
Removing pensive_jennings       ... 
Removing youthful_bartik        ... 
Removing infallible_elion       ... 
Removing nice_hoover            ... 
Removing quirky_cerf            ... 
Removing zen_banach             ... 
Removing stupefied_shirley      ... 
Removing ecstatic_elbakyan      ... 
Removing unruffled_albattani    ... 
Removing gracious_feistel       ... 
Removing quizzical_spence       ... 
Removing sharp_kepler           ... 
Removing flamboyant_bhabha      ... 
Removing amazing_zhukovsky      ... 
Removing eloquent_satoshi       ... 
Removing onpremise_cron_1       ... 
Removing onpremise_worker_1     ... 
Removing onpremise_postgres_1   ... 
Removing onpremise_memcached_1  ... 
Removing onpremise_smtp_1       ... 
Removing onpremise_redis_1      ... 
Removing elastic_galileo        ... done
Removing stupefied_shirley      ... done
Removing exciting_burnell5      ... done
Removing quirky_cerf            ... done
Removing unruffled_albattani    ... done
Removing cranky_volhard         ... done
Removing gracious_feistel       ... done
Removing crazy_brattain         ... done
Removing adoring_chatelet       ... done
Removing ecstatic_elbakyan      ... done
Removing eloquent_satoshi       ... done
Removing wonderful_euclid       ... done
Removing lucid_dubinsky         ... done
Removing blissful_khorana       ... done
Removing quizzical_spence       ... done
Removing flamboyant_bhabha      ... done
Removing youthful_bartik        ... done
Removing nervous_kapitsa        ... done
Removing heuristic_varahamihira ... done
Removing inspiring_curie        ... done
Removing agitated_mclean        ... done
Removing nice_hoover            ... done
Removing amazing_zhukovsky      ... done
Removing eloquent_turing        ... done
Removing lucid_noether          ... done
Removing zen_banach             ... done
Removing flamboyant_babbage     ... done
Removing vigorous_sanderson     ... done
Removing onpremise_worker_1     ... done
Removing elegant_blackburn      ... done
Removing onpremise_redis_1      ... done
Removing onpremise_memcached_1  ... done
Removing infallible_elion       ... done
Removing onpremise_postgres_1   ... done
Removing optimistic_aryabhata   ... done
Removing onpremise_cron_1       ... done
Removing optimistic_kepler      ... done
Removing sharp_kepler           ... done
Removing pensive_jennings       ... done
Removing naughty_pascal         ... done
Removing onpremise_smtp_1       ... done
Removing network onpremise_default
Removing network sentry_onpremise_default
Network sentry_onpremise_default not found.

Creating volumes for persistent storage...
Created sentry-data.
Created sentry-postgres.
Created sentry-redis.
Created sentry-zookeeper.
Created sentry-kafka.
Created sentry-clickhouse.
Created sentry-symbolicator.

Creating sentry/sentry.conf.py...
Creating sentry/config.yml...
Creating sentry/requirements.txt...

Generating secret key...
Secret key written to sentry/config.yml

Fetching and updating Docker images...

latest: Pulling from getsentry/sentry
8d691f585fa8: Pulling fs layer
3fd6980f9df6: Pulling fs layer
c4f890363b98: Pulling fs layer
b8cfb9853b0a: Pulling fs layer
274a8b91d259: Pulling fs layer
b21ea7985fdd: Pulling fs layer
bfa87da1d328: Pulling fs layer
0a8cae89e879: Pulling fs layer
e24cc9ee23b0: Pulling fs layer
ab73045047c1: Pulling fs layer
2ebc2b532a39: Pulling fs layer
aef31dd351c3: Pulling fs layer
e6207e1380cc: Pulling fs layer
39af87d9b2c9: Pulling fs layer
55fc73a43147: Pulling fs layer
2ebc2b532a39: Waiting
aef31dd351c3: Waiting
e6207e1380cc: Waiting
39af87d9b2c9: Waiting
b8cfb9853b0a: Waiting
55fc73a43147: Waiting
274a8b91d259: Waiting
b21ea7985fdd: Waiting
bfa87da1d328: Waiting
0a8cae89e879: Waiting
e24cc9ee23b0: Waiting
ab73045047c1: Waiting
3fd6980f9df6: Verifying Checksum
3fd6980f9df6: Download complete
c4f890363b98: Verifying Checksum
c4f890363b98: Download complete
8d691f585fa8: Verifying Checksum
8d691f585fa8: Download complete
b8cfb9853b0a: Verifying Checksum
b8cfb9853b0a: Download complete
274a8b91d259: Verifying Checksum
274a8b91d259: Download complete
b21ea7985fdd: Verifying Checksum
b21ea7985fdd: Download complete
bfa87da1d328: Verifying Checksum
bfa87da1d328: Download complete
8d691f585fa8: Pull complete
3fd6980f9df6: Pull complete
c4f890363b98: Pull complete
b8cfb9853b0a: Pull complete
274a8b91d259: Pull complete
b21ea7985fdd: Pull complete
bfa87da1d328: Pull complete
ab73045047c1: Verifying Checksum
ab73045047c1: Download complete
2ebc2b532a39: Verifying Checksum
2ebc2b532a39: Download complete
aef31dd351c3: Verifying Checksum
aef31dd351c3: Download complete
e6207e1380cc: Verifying Checksum
e6207e1380cc: Download complete
e24cc9ee23b0: Verifying Checksum
e24cc9ee23b0: Download complete
39af87d9b2c9: Verifying Checksum
39af87d9b2c9: Download complete
0a8cae89e879: Verifying Checksum
0a8cae89e879: Download complete
55fc73a43147: Verifying Checksum
55fc73a43147: Download complete
0a8cae89e879: Pull complete
e24cc9ee23b0: Pull complete
ab73045047c1: Pull complete
2ebc2b532a39: Pull complete
aef31dd351c3: Pull complete
e6207e1380cc: Pull complete
39af87d9b2c9: Pull complete
55fc73a43147: Pull complete
Digest: sha256:fa5254e17dc16d0afc6852d47d0234d6e0ad215db74467a683723200c0b51b68
Status: Downloaded newer image for getsentry/sentry:latest
docker.io/getsentry/sentry:latest

Building and tagging Docker images...

Building web
Step 1/4 : ARG SENTRY_IMAGE
Step 2/4 : FROM ${SENTRY_IMAGE:-getsentry/sentry:latest}
 ---> 9adb7dd2b454
Step 3/4 : COPY . /usr/src/sentry
 ---> d9abff4bd808
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Running in eb0af27071b8
e[91mDEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
e[0mRemoving intermediate container eb0af27071b8
 ---> ecb20dd3ac33

Successfully built ecb20dd3ac33
Successfully tagged sentry-onpremise-local:latest
smtp uses an image, skipping
memcached uses an image, skipping
redis uses an image, skipping
postgres uses an image, skipping
zookeeper uses an image, skipping
kafka uses an image, skipping
clickhouse uses an image, skipping
snuba-api uses an image, skipping
snuba-consumer uses an image, skipping
snuba-outcomes-consumer uses an image, skipping
snuba-replacer uses an image, skipping
symbolicator uses an image, skipping
relay uses an image, skipping
nginx uses an image, skipping
Building snuba-cleanup          ... 
Building symbolicator-cleanup   ... 
Building web                    ... 
Building cron                   ... 
Building worker                 ... 
Building ingest-consumer        ... 
Building post-process-forwarder ... 
Building sentry-cleanup         ... 
Building snuba-cleanup
Building post-process-forwarder
Building sentry-cleanup
Building ingest-consumer
Building worker
Step 1/4 : ARG SENTRY_IMAGE
Step 2/4 : FROM ${SENTRY_IMAGE:-getsentry/sentry:latest}
 ---> 9adb7dd2b454
Step 3/4 : COPY . /usr/src/sentry
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> c7ed8bf14bdd
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Using cache
 ---> d9abff4bd808
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> ecb20dd3ac33

Successfully built ecb20dd3ac33
Step 1/4 : ARG SENTRY_IMAGE
Step 2/4 : FROM ${SENTRY_IMAGE:-getsentry/sentry:latest}
 ---> 9adb7dd2b454
Step 3/4 : COPY . /usr/src/sentry
 ---> Using cache
 ---> d9abff4bd808
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> ecb20dd3ac33

Successfully built ecb20dd3ac33
Successfully tagged sentry-onpremise-local:latest
Building ingest-consumer        ... done
Building web
Successfully tagged sentry-onpremise-local:latest
Building post-process-forwarder ... done
Building symbolicator-cleanup
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> ecb20dd3ac33
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
Step 1/4 : ARG SENTRY_IMAGE
Step 2/4 : FROM ${SENTRY_IMAGE:-getsentry/sentry:latest}
 ---> 9adb7dd2b454
Step 3/4 : COPY . /usr/src/sentry
 ---> Using cache
 ---> d9abff4bd808
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> ecb20dd3ac33

Successfully built ecb20dd3ac33
Successfully tagged sentry-onpremise-local:latest
Building cron
Building worker                 ... done
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> a483d59bf9b5
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Running in c94a73f2d0e3
 ---> Running in f6dc3fde0fec
 ---> Running in 9be497dbf6d4
Step 1/4 : ARG SENTRY_IMAGE
Step 2/4 : FROM ${SENTRY_IMAGE:-getsentry/sentry:latest}
 ---> 9adb7dd2b454
Step 3/4 : COPY . /usr/src/sentry
 ---> Using cache
 ---> d9abff4bd808
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> ecb20dd3ac33

Successfully built ecb20dd3ac33
Successfully tagged sentry-onpremise-local:latest
Building web                    ... done
Step 1/4 : ARG SENTRY_IMAGE
Step 2/4 : FROM ${SENTRY_IMAGE:-getsentry/sentry:latest}
 ---> 9adb7dd2b454
Step 3/4 : COPY . /usr/src/sentry
 ---> Using cache
 ---> d9abff4bd808
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> ecb20dd3ac33

Successfully built ecb20dd3ac33
Successfully tagged sentry-onpremise-local:latest
Building cron                   ... done
Err:1 http://security.debian.org/debian-security stretch/updates InRelease
  Temporary failure resolving 'security.debian.org'
Err:2 http://deb.debian.org/debian stretch InRelease
  Temporary failure resolving 'deb.debian.org'
Err:1 http://deb.debian.org/debian buster InRelease
  Temporary failure resolving 'deb.debian.org'
Err:2 http://security.debian.org/debian-security buster/updates InRelease
  Temporary failure resolving 'security.debian.org'
Err:1 http://deb.debian.org/debian buster InRelease
  Temporary failure resolving 'deb.debian.org'
Err:2 http://security.debian.org/debian-security buster/updates InRelease
  Temporary failure resolving 'security.debian.org'
Err:3 http://deb.debian.org/debian stretch-updates InRelease
  Temporary failure resolving 'deb.debian.org'
Reading package lists...
e[91mW: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch http://security.debian.org/debian-security/dists/stretch/updates/InRelease  Temporary failure resolving 'security.debian.org'
W: Failed to fetch http://deb.debian.org/debian/dists/stretch-updates/InRelease  Temporary failure resolving 'deb.debian.org'
W: Some index files failed to download. They have been ignored, or old ones used instead.
e[0mReading package lists...
Building dependency tree...
Reading state information...
e[91mE: Unable to locate package cron
e[0mRemoving intermediate container 9be497dbf6d4
Err:3 http://deb.debian.org/debian buster-updates InRelease
  Temporary failure resolving 'deb.debian.org'
Reading package lists...
e[91mW: Failed to fetch http://deb.debian.org/debian/dists/buster/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch http://security.debian.org/debian-security/dists/buster/updates/InRelease  Temporary failure resolving 'security.debian.org'
W: Failed to fetch http://deb.debian.org/debian/dists/buster-updates/InRelease  Temporary failure resolving 'deb.debian.org'
W: Some index files failed to download. They have been ignored, or old ones used instead.
e[0mReading package lists...
Building dependency tree...
Reading state information...
e[91mE: Unable to locate package cron
e[0mErr:3 http://deb.debian.org/debian buster-updates InRelease
  Temporary failure resolving 'deb.debian.org'
Reading package lists...
e[91mW: Failed to fetch http://deb.debian.org/debian/dists/buster/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch http://security.debian.org/debian-security/dists/buster/updates/InRelease  Temporary failure resolving 'security.debian.org'
W: Failed to fetch http://deb.debian.org/debian/dists/buster-updates/InRelease  Temporary failure resolving 'deb.debian.org'
W: Some index files failed to download. They have been ignored, or old ones used instead.
e[0mReading package lists...
Building dependency tree...
Reading state information...
e[91mE: Unable to locate package cron
e[0mRemoving intermediate container c94a73f2d0e3
Removing intermediate container f6dc3fde0fec

ERROR: for symbolicator-cleanup  (<Service: symbolicator-cleanup>, "The command '/bin/sh -c apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*' returned a non-zero code: 100")

ERROR: for snuba-cleanup  (<Service: snuba-cleanup>, "The command '/bin/sh -c apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*' returned a non-zero code: 100")

ERROR: for sentry-cleanup  (<Service: sentry-cleanup>, "The command '/bin/sh -c apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*' returned a non-zero code: 100")
Service 'sentry-cleanup' failed to build: The command '/bin/sh -c apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*' returned a non-zero code: 100
Cleaning up...

Can anyone help me upgrading it?
Thanks!
Catalin

2

If you look at the logs you’ll notice that you had an issue connecting to Debian servers when trying to build some of the required images. This may be a temporary networking issue so I’d try again.

If you know you’re behind some kind of a firewall preventing access to certain domains, you’d need to fix that and allow access to Debian servers, at least at the time of the build.

3

Hi BYK, thanks for the info.
However, from the server I can ping the deb.debian.org

[root@sentry ~]# ping deb.debian.org
PING debian.map.fastly.net (151.101.122.133) 56(84) bytes of data.
64 bytes from 151.101.122.133 (151.101.122.133): icmp_seq=1 ttl=58 time=4.16 ms

I think the error cames from the container.

4

Yes the errors come from the container and say

 Temporary failure resolving 'security.debian.org'

So either your Docker network DNS resolver had issues or there’s something else.

1 Like
5

Ok, so it seems indeed there were some DNS problems.
It wasn’t only a DNS problem, but a whole networking one.
Somehow, on the server, the network docker0 interface had 127.0.0.1 IP assigned, which I think it was conflicting with the loopback interface.

I tested network connectivity by using the busybox container and tried to ping 8.8.8.8 from within:

docker run -it --rm busybox
#ping 8.8.8.8

The ping was unreachable.

As a solution, I have assigned another subnet for it, by creating /etc/docker/daemon.json file and restarted the service

{
“bip”: “172.67.0.1/16”
}
systemctl restart docker

Then, testing from within the busybox container, I was able to ping google’s 8.8.8.8 ip.

In the end, I was able to upgrade to sentry 10 by using the install.sh script.

Regards,
Cata

1 Like