Cannot use custom symbol server as issue in writing symcache occurs

JohnClark75 · October 9, 2020, 3:56pm

Hi,

I use sentry on premise
Sentry 20.10.0.dev0b22ca35

Minidumps are uploaded via UE4 key.

We have setup an HTTP symbol server that sentry is using via custom symbol server setting.
Sadly it cannot process the most important file, our game pdb.
We already allocated 16GB to VM as inbetween we saw a demand of around 11GB while processing. Still we get an error when trying to process the symbol file.

In symbolicator log we get:
WARN symbolicator::actors::symcaches > Failed to write symcache: failed to write symcache

We also set our sentry instance to catch symbolicator issues and we get this:

Writing

failed to write symcache

BadDebugFile

malformed debug info file

Pdb

failed to process pdb file

BadObject

invalid pdb file

CrossModuleRefNotFound

Cross module reference 0xb7438fe8 not found in imports

I checked the file and uploaded it via CLI and there it works. Any idea what the issue could be ? Or starting point to look for ?

BYK · October 12, 2020, 10:15am

Symbolicator uses a disk cache through the volume sentry-symbolicator named volume:

github.com

getsentry/onpremise/blob/a9a8b2f98b84bc1de94884dd3226ef3d7d77b29f/docker-compose.yml#L158


  image: snuba-cleanup-onpremise-local
  build:
    context: ./cron
    args:
      BASE_IMAGE: '$SNUBA_IMAGE'
  command: '"*/5 * * * * gosu snuba snuba cleanup --dry-run False"'
symbolicator:
  << : *restart_policy
  image: '$SYMBOLICATOR_IMAGE'
  volumes:
    - 'sentry-symbolicator:/data'
    - type: bind
      read_only: true
      source: ./symbolicator
      target: /etc/symbolicator
  command: run -c /etc/symbolicator/config.yml
symbolicator-cleanup:
  << : *restart_policy
  image: symbolicator-cleanup-onpremise-local
  build:
    context: ./cron

Could you be running out of disk space there somehow?

JohnClark75 · October 12, 2020, 11:56am

This instance itself has around 400GB free. Guess space is not the issue.
The pd_ file (compressed from symstore) is 194MB the original pdb is 1,1GB.
The pd_ is getting downloaded from the symbol server, the symbolicator starts processing and then I get the error message…
Still irritated about the CrossModuleRefNotFound message as symbolicator has full access to any other symbols on the server and as described it’s working out when using the CLI to upload the PDB directly.

JohnClark75 · October 13, 2020, 1:03pm

is there any test for the symcache so I can verify the functionality is working and it is not related to file size?

are there settings around the symcache functionality I could check?

JohnClark75 · October 13, 2020, 4:46pm

After killing all container and volumes and a reinstall it works now.
Only difference is that I changed the read only flag for the symbolicator volume in the image section image to false!?

BYK · October 14, 2020, 9:06am

The readonly flag was only for the configuration which should be read-only. Can you share your exact change (maybe as a patch file?)

JohnClark75 · October 14, 2020, 9:46am

only change was the volume flag…
sadly while first conversion went well getting issues now again. again the pdb file is usable when opening the dump with VS (symbol setup in VS is pointing to the same internal symbol server).

Writing

failed to write symcache

BadDebugFile

malformed debug info file

Pdb

failed to process pdb file

BadObject

invalid pdb file

TypeNotFound

Type 455403655 not found

What are the main differences between using custom symbol server and using the CLI?
Would expect after symbol is uploaded (CLI) or pulled from symbol server (custom symbol server) it should behave the same…

jauer · October 14, 2020, 12:02pm

Whether you use a custom symbol server or the CLI has no effect on how the file is processed in Sentry. In the end, the Symbolicator service always retrieves and unpacks the file and then runs the same logic on it. This does not happen when you upload the file, but when it symbolicates a crash that requires the file.

The errors you’re receiving ultimately have to do with how we process the file. The fact that you’re able to open the Minidump in VS indicates a potential bug in our PDB file handling. In order to debug this, we would need access to the very file that is causing the issue, however. Would you be able to provide one of the offending PDBs so we can analyze them?

Do you read these errors from logs, or from internal reporting to Sentry? Depending on this, there are different steps on how you identify the file that is causing these errors.

JohnClark75 · October 14, 2020, 2:04pm

I will check if I can share the PDB.

ATM I do both. So reading the log from the symbolicator container and also getting the errors in sentry as well.

The main question for me is why is the exact same PDB is processed fine when coming from CLI while the same PD_ (just the compressed one) is not…

Did that several times now when the PD_ was not working … unpacking it and uploading via CLI and it was processed fine.

So if processing the PDB is the same regardless CLI or custom symbol server I would think the issue is somewhere within the unpacking of the PD_ …

jauer · October 22, 2020, 7:19pm

You’re right. I had somehow missed your note on compression in the prior messages.

Which compression is used for the PD_ file? We support Gzip, zlib (both with and without header), Zstandard (1.4.4), and Cabinet (CAB). Potentially, our symbolication service is not able to handle the compression format.

JohnClark75 · October 22, 2020, 9:32pm

The weird thing is the issue is happening approx 3 out of 10. So we have symbols coming from the same build system and they work fine.
We use MSCF so cab.