Sentry-cli source map artifacts upload throws 500 error

[vinnie]

I am running on-premise sentry (version 9.1.2) in a kubernetes cluster. Lately, I am seeing a 500 error every time I attempt to create a new release and upload source maps to any project in sentry. When I check the release, I do not see any artifacts uploaded.

This is the DEBUG log from sentry-cli during the upload:-

  DEBUG   2021-03-25 16:46:02.339134 +0000 body: {"detail": "Internal Error", "errorId": "9710736c7fea44c4a28174afe87598d5"}
  DEBUG   2021-03-25 16:46:02.435353 +0000 < HTTP/1.1 500 Internal Server Error
  DEBUG   2021-03-25 16:46:02.435453 +0000 < Date: Thu, 25 Mar 2021 16:46:02
  DEBUG   2021-03-25 16:46:02.435507 +0000 < Content-Type: application/json
  DEBUG   2021-03-25 16:46:02.435544 +0000 < Content-Length: 75
  DEBUG   2021-03-25 16:46:02.435649 +0000 < Connection: keep-alive
  DEBUG   2021-03-25 16:46:02.435977 +0000 < Allow: GET, POST, HEAD, OPTIONS
  DEBUG   2021-03-25 16:46:02.436066 +0000 < Content-Language: en
  DEBUG   2021-03-25 16:46:02.436115 +0000 < Strict-Transport-Security: max-age=315360000; includeSubDomains
  DEBUG   2021-03-25 16:46:02.436175 +0000 < Vary: Accept-Language, Cookie
  DEBUG   2021-03-25 16:46:02.436288 +0000 < X-Content-Type-Options: nosniff
  DEBUG   2021-03-25 16:46:02.436334 +0000 < X-Frame-Options: deny
  DEBUG   2021-03-25 16:46:02.436479 +0000 < X-Xss-Protection: 1; mode=block
  DEBUG   2021-03-25 16:46:02.436565 +0000 response status: 500
  DEBUG   2021-03-25 16:46:02.436665 +0000 body: {"detail": "Internal Error", "errorId": "aa8bb8da822c4feda1cf4859045e024e"}
error: API request failed
  caused by: sentry reported an error: request failure (http status: 500)
  Object({"detail": String("Internal Error"), "errorId": String("414e2adb4f5c4666991731bb53c15bd5")})
  DEBUG   2021-03-25 16:46:02.441432 +0000 client close; no transport to shut down  (from sentry)

Checked sentry web logs and all I could see was :-

"10.10.2.16 - - [25/Mar/2021:16:46:02 +0000] \"POST /api/0/projects/projectA/projectA-project/releases/test2/files/ HTTP/1.1\" 500 340 \"-\" \"sentry-cli/1.61.0\""

The filestore is an aws s3 bucket (the auth & privileges works just fine) and I can’t seem to figure the heads and tails of this.

Is there any way we can increase the verbosity of the sentry server logs ?

[BYK]

It shows you the error id that you can look up on the “internal” project on your Sentry instance to get more details: 414e2adb4f5c4666991731bb53c15bd5

[vinnie]

Hi BYK,

These error IDs never show up on the sentry web logs. All I can see is the API POST call failure with the error code 500. I can see the release created in the project, but not even a single artifact gets uploaded.
This is not specific to one internal project. This is now happening for all projects. Is there a way to enable verbose logging in sentry web to capture the root cause of these API request failures?

[maxence]

Hello !

We encountered this very issue today when setting up Sentry on premise v21.03. Here is the log we retrieved from the internal project on our sentry application.

Our filestore configuration is as follows :

filestore:
backend: s3
s3:
accessKey: accessKey
secretKey: secretKey
bucketName: sentry-bucket
endpointUrl: https://s3.fr-par.scw.cloud
region_name: fr-par
signature_version: s3v4

As you can see we use a filestorage provided by Scaleway.

We obtain this issue by performing the same operation as vinnie described :

sentry-cli releases files upload-sourcemaps ./dist/maps

Hopefully it will help you unblocking us

Edit : additional context details

[maxence]

Hello,

Any advice on this ? We still have the issue on our installation

[BYK]

@maxence if you are sure your settings are correct (regarding signing keys and credentials) I’d check if the time is correctly set on your Sentry instance as some signing methods rely on correct & synced clocks.

[vinnie]

Hi BYK,

Do you have any suggestions for me? The error id doesn’t show up in logs, so there is no way to identify the root cause of the problem.

[BYK]

@vinnie there must be more details on the logs for sure. You may wanna try turning on more logging by following: https://develop.sentry.dev/config/#logging

[vinnie]

@BYK I have tried to enable verbose logging by adding this line in /etc/sentry/sentry.conf.py

LOGGING['default_level'] = 'DEBUG'

The pod fails to launch and it just spits out the error:-

!! Configuration error: NameError: name 'LOGGING' is not defined

[BYK]

Do you have this line at the top of that file: from sentry.conf.server import *

[vinnie]

I did not have that in the configuration file. I added it now and I was able to schedule the pod. Thanks! I will check the logs and see what is going on under the hood.