Cannot use custom symbol server as issue in writing symcache occurs

On-Premise
#1

Hi,

I use sentry on premise
Sentry 20.10.0.dev0b22ca35

Minidumps are uploaded via UE4 key.

We have setup an HTTP symbol server that sentry is using via custom symbol server setting.
Sadly it cannot process the most important file, our game pdb.
We already allocated 16GB to VM as inbetween we saw a demand of around 11GB while processing. Still we get an error when trying to process the symbol file.

In symbolicator log we get:
WARN symbolicator::actors::symcaches > Failed to write symcache: failed to write symcache

We also set our sentry instance to catch symbolicator issues and we get this:

Writing

failed to write symcache

BadDebugFile

malformed debug info file

Pdb

failed to process pdb file

BadObject

invalid pdb file

CrossModuleRefNotFound

Cross module reference 0xb7438fe8 not found in imports

I checked the file and uploaded it via CLI and there it works. Any idea what the issue could be ? Or starting point to look for ?

#2

Symbolicator uses a disk cache through the volume sentry-symbolicator named volume:

Could you be running out of disk space there somehow?

#3

This instance itself has around 400GB free. Guess space is not the issue.
The pd_ file (compressed from symstore) is 194MB the original pdb is 1,1GB.
The pd_ is getting downloaded from the symbol server, the symbolicator starts processing and then I get the error message…
Still irritated about the CrossModuleRefNotFound message as symbolicator has full access to any other symbols on the server and as described it’s working out when using the CLI to upload the PDB directly.

#4

is there any test for the symcache so I can verify the functionality is working and it is not related to file size?

are there settings around the symcache functionality I could check?

#5

After killing all container and volumes and a reinstall it works now.
Only difference is that I changed the read only flag for the symbolicator volume in the image section image to false!?

#6

The readonly flag was only for the configuration which should be read-only. Can you share your exact change (maybe as a patch file?)

#7

only change was the volume flag…
sadly while first conversion went well getting issues now again. again the pdb file is usable when opening the dump with VS (symbol setup in VS is pointing to the same internal symbol server).

Writing

failed to write symcache

BadDebugFile

malformed debug info file

Pdb

failed to process pdb file

BadObject

invalid pdb file

TypeNotFound

Type 455403655 not found

What are the main differences between using custom symbol server and using the CLI?
Would expect after symbol is uploaded (CLI) or pulled from symbol server (custom symbol server) it should behave the same…

#8

Whether you use a custom symbol server or the CLI has no effect on how the file is processed in Sentry. In the end, the Symbolicator service always retrieves and unpacks the file and then runs the same logic on it. This does not happen when you upload the file, but when it symbolicates a crash that requires the file.

The errors you’re receiving ultimately have to do with how we process the file. The fact that you’re able to open the Minidump in VS indicates a potential bug in our PDB file handling. In order to debug this, we would need access to the very file that is causing the issue, however. Would you be able to provide one of the offending PDBs so we can analyze them?

Do you read these errors from logs, or from internal reporting to Sentry? Depending on this, there are different steps on how you identify the file that is causing these errors.

1 Like
#9

I will check if I can share the PDB.

ATM I do both. So reading the log from the symbolicator container and also getting the errors in sentry as well.

The main question for me is why is the exact same PDB is processed fine when coming from CLI while the same PD_ (just the compressed one) is not…

Did that several times now when the PD_ was not working … unpacking it and uploading via CLI and it was processed fine.

So if processing the PDB is the same regardless CLI or custom symbol server I would think the issue is somewhere within the unpacking of the PD_ …

#10

You’re right. I had somehow missed your note on compression in the prior messages.

Which compression is used for the PD_ file? We support Gzip, zlib (both with and without header), Zstandard (1.4.4), and Cabinet (CAB). Potentially, our symbolication service is not able to handle the compression format.

#11

The weird thing is the issue is happening approx 3 out of 10. So we have symbols coming from the same build system and they work fine.
We use MSCF so cab.

#12

Thanks. Any chance you could mail me one of the files that fails consistently? I’d like to check if our cab extractor is the culprit.

#13

Hi,
as I wrote I could upload them via CLI and they worked… also added second project and they worked there as well. So seems to be related to something around the unpacking/create cache step.
ATM problem is solved as I booked attachments for our SaaS (attachments were only reason for us to setup on premise)

#14

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.