Uploading Source Files with Debug Information

tom-sonnox · October 14, 2019, 11:13am

We’re currently testing out On-Premise and would like to see source code along with the stack trace for minidump type projects. After some digging in the documentation this seems possible, but our attempts to upload source files along with debug information don’t work, with the sentry-cli command emitting a warning:

CLI command is as follows:

$ sentry-cli upload-dif --include-sources --project [project-name] [search-path]

Project type is ‘minidump’. We’re using On-Premise Sentry from the v10 branch.

The output from the API for <...>/api/0/organizations/sentry/chunk-upload/ is:

{
    "maxFileSize": 2147483648,
    "hashAlgorithm": "sha1",
    "concurrency": 8,
    "chunkSize": 8388608,
    "maxRequestSize": 33554432,
    "url": "<...>/api/0/organizations/sentry/chunk-upload/",
    "chunksPerRequest": 64,
    "accept": [
        "debug_files",
        "release_files",
        "pdbs"
    ],
    "compression": [
        "gzip"
    ]
}

Where, if the chunk.py file in the main sentry repository is anything to go by, I would expect to also see "sources" in the accept list.

Is there a configuration option that needs to be set to allow this?

BYK · October 14, 2019, 12:04pm

Are you sure your sentry-cli is configured to use this new server?

Btw. the v10 branch is not fully supported yet and may incur breaking changes during its development so beware

tom-sonnox · October 14, 2019, 1:30pm

I’ve just double-checked the SENTRY_URL environment variable, and it is pointing to the correct server, if that’s what you mean? The version of sentry-cli we’re using is 1.48.0.

OK. The reason we’re on that branch is that we were seeing this same issue when trying to upload source files using the stable release of On-Premise. We thought that maybe the --include-sources CLI option was only for use with v10.

Do you expect this feature to work with the stable release?

tom-sonnox · October 14, 2019, 2:55pm

FYI, I think support for this was added when this was merged:

BYK · October 14, 2019, 7:03pm

Yeah, that’s what I meant and it sounds about right. Maybe you can try using the verbose output and share it here so we can try debugging from there?

This feature is only available on master (hence v10) as far as I know. The symbolicator integration on the v10 branch was a bit spotty in the earlier days. Are you using the very latest version of this branch? Can you also try removing all the old config files and use the provided .example ones from the latest version of that branch?

tom-sonnox · October 15, 2019, 9:56am

Thanks for your help on this.

Here’s the output with --log-level TRACE up to the warning about source uploads (the rest of it is related to the debug information files).

INFO    2019-10-15 08:55:19.142706 +01:00 Loaded config from ~/.sentryclirc
DEBUG   2019-10-15 08:55:19.142951 +01:00 sentry-cli version: 1.48.0, platform: "darwin", architecture: "x86_64"
INFO    2019-10-15 08:55:19.142977 +01:00 sentry-cli was invoked with the following command line: "sentry-cli" "--log-level" "TRACE" "upload-dif" "--include-sources" "--project" "test" "~/dev/test"
DEBUG   2019-10-15 08:55:19.144711 +01:00 request GET <...>/api/0/organizations/sonnox/chunk-upload/
DEBUG   2019-10-15 08:55:19.144755 +01:00 using token authentication
DEBUG   2019-10-15 08:55:19.144769 +01:00 retry number 0, max retries: 0
DEBUG   2019-10-15 08:55:19.151783 +01:00 > GET /api/0/organizations/sonnox/chunk-upload/ HTTP/1.1
DEBUG   2019-10-15 08:55:19.151812 +01:00 > Host: <...>
DEBUG   2019-10-15 08:55:19.151819 +01:00 > Accept: */*
DEBUG   2019-10-15 08:55:19.151824 +01:00 > Connection: TE
DEBUG   2019-10-15 08:55:19.151831 +01:00 > TE: gzip
DEBUG   2019-10-15 08:55:19.151835 +01:00 > User-Agent: sentry-cli/1.48.0
DEBUG   2019-10-15 08:55:19.152357 +01:00 > Authorization: Bearer ***********
DEBUG   2019-10-15 08:55:19.202291 +01:00 < HTTP/1.1 200 OK
DEBUG   2019-10-15 08:55:19.202376 +01:00 < Content-Length: 285
DEBUG   2019-10-15 08:55:19.202407 +01:00 < X-XSS-Protection: 1; mode=block
DEBUG   2019-10-15 08:55:19.202428 +01:00 < Content-Language: en
DEBUG   2019-10-15 08:55:19.202445 +01:00 < X-Content-Type-Options: nosniff
DEBUG   2019-10-15 08:55:19.202462 +01:00 < Vary: Accept-Language, Cookie
DEBUG   2019-10-15 08:55:19.202478 +01:00 < Allow: GET, POST, HEAD, OPTIONS
DEBUG   2019-10-15 08:55:19.202491 +01:00 < X-Frame-Options: deny
DEBUG   2019-10-15 08:55:19.202506 +01:00 < Content-Type: application/json
DEBUG   2019-10-15 08:55:19.202644 +01:00 response status: 200
DEBUG   2019-10-15 08:55:19.202671 +01:00 body: {"maxFileSize":2147483648,"hashAlgorithm":"sha1","concurrency":8,"chunkSize":8388608,"maxRequestSize":33554432,"url":"<...>/api/0/organizations/sonnox/chunk-upload/","chunksPerRequest":64,"accept":["debug_files","release_files","pdbs"],"compression":["gzip"]}
WARN    2019-10-15 08:55:19.202731 +01:00 Source uploads are not supported by the configured Sentry server

As you can see, the request response shows that "sources" is missing from the accept list.

Our local onpremise repository is now pointed at ca097228. We completely removed the repo, along with all configuration files, did a fresh clone, then did a clean install (removing all docker containers, images and volumes prior to installation). Unfortunately the issue is still there.

One thing to note is, after the clean install, I decided to check the server for the chunk.py file to see if it matched up with the one on the sentry master branch. The following:

find / -wholename */sentry/api/endpoints/chunk.py

returns a few matching files, but none of them match the one on GitHub. They’re all missing "sources" from here:

github.com

getsentry/sentry/blob/196a41f8cffa4f5f0784a9638f544b8cb034aaba/src/sentry/api/endpoints/chunk.py#L27


      
          from sentry.utils.files import get_max_file_size
          
          
          # The blob size must be a power of two
          CHUNK_UPLOAD_BLOB_SIZE = 8 * 1024 * 1024  # 8MB
          MAX_CHUNKS_PER_REQUEST = 64
          MAX_REQUEST_SIZE = 32 * 1024 * 1024
          MAX_CONCURRENCY = settings.DEBUG and 1 or 8
          HASH_ALGORITHM = "sha1"
          
          CHUNK_UPLOAD_ACCEPT = (
              "debug_files",  # DIF assemble
              "release_files",  # Release files assemble
              "pdbs",  # PDB upload and debug id override
              "sources",  # Source artifact bundle upload
          )
          
          
          class GzipChunk(BytesIO):
              def __init__(self, file):
                  data = GzipFile(fileobj=file, mode="rb").read()

This makes me think that the Sentry docker image being pulled during ./install.sh isn’t using the correct/latest commit. Could that be the case? Is there any way to check this?

BYK · October 15, 2019, 4:38pm

Ah, maybe it is because you don’t pull the image again? We don’t force pulling during docker-compose build steps so that might be it.

Would you mind trying docker rmi -f getsentry/sentry:latest and installing again?

tom-sonnox · October 16, 2019, 9:18am

So after a clean install, I ran that docker command but the output was:

$ docker rmi -f getsentry/sentry:latest
Error: No such image: getsentry/sentry:latest

This is because the actual image pulled during the onpremise install is latest-onbuild:

github.com

getsentry/self-hosted/blob/ca097228a957bc4f9c4922a261e358e8d57716dd/sentry/Dockerfile#L1


      
          FROM ${SENTRY_IMAGE:-getsentry/sentry:latest}-onbuild

As a test, I tried another clean install, but removed the -onbuild suffix from that line in the dockerfile before installing so that the image actually pulled would be getsentry/sentry:latest.

Installation didn’t complete, it failed with this error:

!! Configuration error: Exception: Error: REDIS_PORT_6379_TCP_ADDR (or SENTRY_REDIS_HOST) is undefined, did you forget to `--link` a redis container?

But more importantly, I checked the chunk.py file installed using this image of sentry and it does match the one linked to above, and contains the expected "sources" line.

Why doesn’t the getsentry/sentry:latest-onbuild image use the same Sentry repo commit as the getsentry/sentry:latest image?

Is there a way to use make the getsentry/sentry:latest image work with onpremise?

BYK · October 16, 2019, 6:26pm

Hey, sorry for the confusion - I totally meant docker rmi -f getsentry/sentry:latest-onbuild. You need the onbuild image for it to copy in your custom configuration. It is already based on getsentry/sentry:latest image anyway so once you remove that and pull the latest, it should fix the issue.

tom-sonnox · October 17, 2019, 9:11am

OK, I had tried that. I can confirm that I am pulling the latest docker image available.

To make sure I was actually getting the correct image, I matched up the digest sha to the one on Docker Hub.

This is the relevant section of output from running ./install.sh on our server:

Building web
Step 1/1 : FROM ${SENTRY_IMAGE:-getsentry/sentry:latest}-onbuild
latest-onbuild: Pulling from getsentry/sentry
b8f262c62ec6: Already exists
8cbb51e0b077: Pull complete
82627a456962: Pull complete
33f3f5c560fe: Pull complete
dce740aa2980: Pull complete
a8d6a008e5a9: Pull complete
0da56cd0087d: Pull complete
26985a668963: Pull complete
573e67692cef: Pull complete
e9ff2b1b38a3: Pull complete
Digest: sha256:e43e4ce7451c1248f2ce876a922f5e43c928844dfaa2c0d099305c94280a0365
Status: Downloaded newer image for getsentry/sentry:latest-onbuild

The Digest: sha256: matches up with what was the latest at time of running the script.

Therefore I am 100% certain that the latest docker image is being pulled.

Now, I decided to look more closely at those Docker images, comparing the latest-onbuild to latest and something just doesn’t seem right. It seems to me that the commands for latest and latest-onbuild should be the same apart from some extra steps for configuration (as you mentioned before) but the command history for latest-onbuild hasn’t changed for quite some time.

In fact, the dockerfile command history for this image from a month ago, and this image from today seem to be exactly the same.

That just doesn’t seem correct. Could you take a look please?

tom-sonnox · October 17, 2019, 4:31pm

I think this line here:

github.com

getsentry/sentry/blob/fa5b03ca360911f3c4ee8d6514fc4ca03daf1456/docker/onbuild/Dockerfile#L1


FROM getsentry/sentry:git


WORKDIR /usr/src/sentry


# Add WORKDIR to PYTHONPATH so local python files don't need to be installed
ENV PYTHONPATH /usr/src/sentry
ONBUILD COPY . /usr/src/sentry


# Hook for installing additional plugins
ONBUILD RUN if [ -s requirements.txt ]; then pip install -r requirements.txt; fi

should be getsentry/sentry:latest to get the latest stuff from master. Is that right?

tom-sonnox · October 17, 2019, 4:54pm

BYK · October 17, 2019, 4:57pm

Ah, that’s what I feared and I apologize for the inconvenience. This is due to very erratic behavior on Docker Hub autobuilds and I’m moving all those to Google Cloud Build at the moment.

Can you try with SENTRY_IMAGE=us.gcr.io/sentryio/sentry:b662e53 and see if it works now? This is the image I built off of my branch which is up-to-date with master. We’ll start pushing these images tagged with :latest to Docker Hub under getsentry/sentry soon too.

BYK · October 17, 2019, 5:12pm

Hey, thanks a lot for that fix, totally missed that

I merged that into my branch too and you can use SENTRY_IMAGE=us.gcr.io/sentryio/sentry:3da2b67 once it is done. I’ve also fixed a SENTRY_IMAGE related bug on the v10 branch so make sure to get its latest version.

Thanks a lot again for all your help and patience!

tom-sonnox · October 17, 2019, 5:14pm

I got source uploads working by:

Removing the existing docker images for Sentry:

docker rmi -f getsentry/sentry:latest-onbuild
docker rmi -f sentry-onpremise-local:latest

Cloning the main Sentry repository and making the modification to docker/onbuild/Dockerfile I mentioned above:

cd ~/
git clone https://github.com/getsentry/sentry.git
cd ./sentry
vi ./docker/onbuild/Dockerfile

Building the docker image with:

docker build -t "getsentry/sentry:latest-onbuild" -f "./docker/onbuild/Dockerfile" .

Then running the install again:

cd ~/onpremise
./install.sh

Everything is working as expected now

I’ll try your fix out too!

Topic		Replies	Views
Ability to upload Source for any language Feedback	1	1293	April 6, 2017
Sentry-cli source map artifacts upload throws 500 error On-Premise	11	2480	August 22, 2021
Sourcemap upload failing when file size more than 20MB On-Premise	1	6101	December 8, 2018
Upload source map files successfully, but there are no artifacts in this archive Feedback	0	1341	December 3, 2020
OnPremiseSentry: S3 multipart upload is failing	0	603	March 9, 2021

Uploading Source Files with Debug Information

Related Topics