After migration 10.1.0 to 21.5.1 internal errors FileNotFoundError /avatar/{avatar_id}/

On-Premise
1

Hello,

A few days ago I have made migration our on-premise Sentry from 10.1.0 to 20.10.1 and next to 21.5.1. Now I got issues in internal project:

FileNotFoundError */avatar/{avatar_id}/*

[Errno 2] No such file or directory: '/data/files/18043/18828/3833d5c8447e46b6ae12e296c163597c'

How can i fix this?

2

Sounds like you have data or permission issues with your sentry-data volume.

You need these changes fix(data): Use `/data` for persisting files and file caches by BYK · Pull Request #295 · getsentry/onpremise · GitHub

3

Hi @BYK

Thanks for quick answer. I have checked the files and:

docker-compose.yml - has correct form
sentry/config.yml - has correct form

but my install.sh has:

#!/usr/bin/env bash
set -e
if [[ -n "$MSYSTEM" ]]; then
  echo "Seems like you are using an MSYS2-based system (such as Git Bash) which is not supported. Please use WSL instead.";
  exit 1
fi

source "$(dirname $0)/install/_lib.sh"  # does a `cd .../install/`, among other things

source parse-cli.sh
source error-handling.sh
source check-minimum-requirements.sh
source create-docker-volumes.sh
source ensure-files-from-examples.sh
source generate-secret-key.sh
source replace-tsdb.sh
source update-docker-images.sh
source build-docker-images.sh
source turn-things-off.sh
source set-up-zookeeper.sh
source bootstrap-snuba.sh
source create-kafka-topics.sh
source upgrade-postgres.sh
source set-up-and-migrate-database.sh
source migrate-file-storage.sh
source relay-credentials.sh
source geoip.sh
source wrap-up.sh

Where should I add the migration part, on beginning, or end, or somewhere in de middle?

4

Your install.sh file is fine. This means it already has the patch in.

What do you mean by “correct form”. Have you checked those variables?

Also can you try running the following manually:

docker run --rm -it -v sentry-data:/data alpine ash -c "mkdir -p /tmp/files; mv /data/* /tmp/files/; mv /tmp/files /data/files"
5

This means that I have these changes in both files:

sentry/config.yml

################
# File storage #
################

# Uploaded media uses these `filestore` settings. The available
# backends are either `filesystem` or `s3`.

filestore.backend: 'filesystem'
filestore.options:
  location: '/data/files'
dsym.cache-path: '/data/dsym-cache'
releasefile.cache-path: '/data/releasefile-cache'

In docker-compouse.yml

version: "3.4"
x-restart-policy: &restart_policy
  restart: unless-stopped
x-sentry-defaults: &sentry_defaults
  <<: *restart_policy
  image: "$SENTRY_IMAGE"
  depends_on:
    - redis
    - postgres
    - memcached
    - smtp
    - snuba-api
    - snuba-consumer
    - snuba-outcomes-consumer
    - snuba-sessions-consumer
    - snuba-transactions-consumer
    - snuba-subscription-consumer-events
    - snuba-subscription-consumer-transactions
    - snuba-replacer
    - symbolicator
    - kafka
  entrypoint: "/etc/sentry/entrypoint.sh"
  command: ["run", "web"]
  environment:
    PYTHONUSERBASE: "/data/custom-packages"
    SENTRY_CONF: "/etc/sentry"
    SNUBA: "http://snuba-api:1218"
    # Leaving the value empty to just pass whatever is set
    # on the host system (or in the .env file)
    SENTRY_EVENT_RETENTION_DAYS:
  volumes:
    - "sentry-data:/data"
    - "./sentry:/etc/sentry"
    - "./geoip:/geoip:ro"

I will try run the migration manually, as you wrote

1 Like
6

Hi @BYK

Manually migration didn’t help. I got the error for the same file all time. Can I manually create the file?

7

You cannot manually create these files as they are user-uploaded avatars. You may try uploading your (or whomever’s) avatar again to make it go away.

Btw, have you restarted your Sentry instances (web, worker, cron) after the manual migration?

8

Yes, I restarted the Sentry instance after migration.

How can I find, to whom belongs, the problematic avatar?

9

FWIW - I don’t know if this works on Sentry 10.x, but the steps below worked for me coming from Sentry 9.0.0.

This was indeed a bit of an unexpected nuisance, because the path which points to uploaded media differs from 9.1.2 and later versions:

First, I hope you have the old Sentry 10 around, if not it’s game over…

Copy the files from inside sentry_web container to the local filesystem. Here’s the idea.
As user root this is what I did on Sentry 9.0.0:

mkdir ~/tmp
cd ~/tmp
docker cp sentry_web_1:/var/lib/sentry/files/ .

Since Sentry 10.0.1 seems to have its datafiles set to location: '/data/files', you might need to use a different path for that Docker copy command:

docker cp sentry_web_1:/data/files/ .

(Also it might be the web contrainer is named sentry_onpremise_web_1 in Sentry 10.x so you might need to change to that name.)

When the above step finished successfully, create a tarball that can be (secure) copied to the new Sentry host:

tar zcf sentry_data.tgz files/

On the NEW Sentry host unpack the tarball, using the same path /root/tmp/.

cd /root/tmp/
tar zxf sentry_data.tgz

Now these files can be copied into one of the Sentry 21.x containers. I copied them into sentry_onpremise_worker_1:

cd files
docker cp . sentry_onpremise_worker_1:/data/files/
docker exec -ti sentry_onpremise_web_1 bash
chown -R sentry:sentry /data/files

As said, this did the trick for me, but your mileage may vary.

10

Hi @hbokh

Thank you for your answer, but I don’t have any copies of Sentry 10.1 anymore :cry:

11

I’m surprised that you need this step as we try to do this automatically:

What are we missing?

12

I guess you’re missing nothing. @BYK. I “moved” Sentry to a new host: a set up with a Postgres dump from v9.0.0.

1 Like
13

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.