I’ve created a custom client for posting native errors to sentry that includes stack traces. How do I get symbolication to run on these event stack traces? I’m uploading the debug information files with sentry-cli successfully and sentry is parsing the stack traces successfully. However, I don’t see how sentry can correlate the instuction_addrs that I’m including to specific debug information files. I see that the list of debug information files from the settings page has the build id that I could include in the stack trace json, but I don’t see anything in the docs on how I would include that.
In NativeStacktraceProcessor, it seems that sentry is calling out to symbolic to determine what object to use (ObjectLookup). I naively tried just using obj_uuid as a field name in the json, but that didn’t work.
Not much I can add here, but the performance hit on a serious error should be insignificant.
This may or may not be relevant, but our symbolification stuff also only works on known formats, and we don’t support all types of binary formats yet even with said formats.
It might help to provide more information about the runtime and distribution.
We’re generating stack traces on a more frequent basis than just serious errors. We often want to generate the stack trace before deciding if it’s worth sending to Sentry.
What I’m doing is pretty standard. I’m using gcc 8.1.0 and extracting the debug information using objcopy --only-keep-debug and uploading it using sentry-cli. I don’t think formats are the issue. I think the issue is just my ignorance of some of the details of what fields I should populating with what when I send events.
I get stack trace information using backtrace() on Linux and CaptureStackBackTrace() on Windows. I add the stack trace information to the event based on this: https://docs.sentry.io/clientdev/interfaces/stacktrace/?platform=javascript . It seems that one thing I missed is that I need to include the debug_meta information as well, so I’ve been working on generating that, but I’m not quite sure I’m getting that right. I’m mostly basing it on some of the same json files in the sentry source, but those all look like they’re coming from Macs, so I’m not sure I’m doing it right.
At this point, I would just welcome some insight into how the whole thing works and what fields I need to make sure I populate to get Sentry to attempt native symbolication because I’m not quite sure how that works. Even if I had some correctly formatted sample json files from a Linux or Windows machine, I think that would help.
Added "platform":"native" to each frame which seems to have triggered symbolication, but I just get <unknown> 0x7f91f506c445 <unknown>, which is progress.
I just struggled with this too, and couldn’t find a solution. It looks like Sentry lets you upload ELF debug info, but not on a per-release basis. Instead, it associates crashdumps with the correct version of the ELF file based on the “build ID” – which the docs falsely claims “all recent compilers add automatically” – but there’s no way to specify the build ID to use for a particular event. I guess this can only really be used for symbolicating crash dumps in some special format.
I could be wrong about all this, though. The docs aren’t very clear.
I ultimately ended up solving my problem by shelling out to addr2line to do symbolication before upload. This requires shipping the debug symbols along with the binary but I don’t have a problem with that for my use case.
GCC adds it, but I’m not sure about other compilers. I was able to get all the information I needed but I was never able to get Sentry to actually use the debug files that I uploaded and I ran out of patience with the whole thing, especially given the lack of response on this forum. It’s frustrating because I’m sure there’s someone at Sentry that could literally answer this question in about five minutes.
As @zeeg has mentioned, we can currently only provide first-class support for minidumps as documented. The performance overhead of generating a minidump vs manually creating the event payload is negligable, especially since Breakpad and Crashpad can be configured to emit insanely small dumps.
To create a custom payload that will work fine with Sentry I would suggest to look at lang/native/minidump.py. However, here’s a couple of points to consider:
The event’s platform needs to be set to "native". Alternatively, this can be set on every frame but I would not recommend that.
We assign frames have a "trust" attribute which you can safely omit, since you are stackwalking in the running process.
Each frame needs a instruction_addr and currently also a function which you should hard-code to "<unknown>". The latter requirement will be dropped in near future.
You need to list all loaded libraries / modules. Their type must be "symbolic" and they must contain a valid "id". Do not add items to this list without an ID (more on that below). Also, images need an image_addr and image_size.
Since memory addresses like instruction_addr or image_addr can hold values larger than the guaranteed JSON number precision, encode them as hex string (e.g. "0xfeedface").
The crashing stack trace must go inside an exception. All other stack traces should go inside the threads list. Only these stack traces will be symbolicated.
Optionally, set a package on the frames with the name of the library/image. Do not use module for that, it has different semantics.
Now, regarding debug identifiers, we are using the same as Google Breakpad. While our canonical formatting for these IDs is lowercase and with dashes, you can also do uppercase and without dashes, though. Depending on what your platform is, you need to employ a different strategy to resolve these identifiers:
On Windows, take the GUID and Age from the CodeView PDB70 debug information record and format them in proper LE byteorder.
On MacOS, directly use the value of the LC_UUID load command formatted as hex.
On Linux, this is more complicated. As documentation states, “recent compilers” should generate a build ID, specifically the .note.gnu.build-id containing a 20+ byte hash. As it turns out, we were wrong and lld needs the additional --build-id flag to emit this (pass as -Wl,--build-id to clang). Even though there is a fallback implemented, I would highly recommend to set this flag. Then, take the first 16 bytes of that hash, reinterpret it as little-endian GUID, and format it out. Here’s what we do in our Rust implementation:
data[0..16].copy_from_slice(&identifier[0..16]);
if little_endian {
// The file ELF file targets a little endian architecture. Convert to
// network byte order (big endian) to match the Breakpad processor's
// expectations. For big endian object files, this is not needed.
data[0..4].reverse(); // uuid field 1
data[4..6].reverse(); // uuid field 2
data[6..8].reverse(); // uuid field 3
}
Again, we do this to be compatible with Breakpad’s minidumps. Finally, use sentry-cli upload-dif to upload the raw symbols.
We are aware that documentation is not ideal around manually creating these payloads. We are still fiddling with the details and reserve to make internally breaking changes to the processing pipeline. Once everything is stable, we can consider to add more documentation around that.
This obviously does not apply to the missing --build-id flag regarding the clang linker. We will add this to the documentation ASAP. Sorry for this oversight!