I’m experiencing an issue on a fresh installation of Sentry 10, using the automated onpremise setup on an Ubuntu 18.04 machine. I’ve set up a “minidump” project and configured my crashpad integration to send minidumps to it (for a x86 Windows app). I have also specified a custom in-house symbol server for Sentry to use when looking up symbol files. However, even though the minidump submission and analysis seems to work as expected, Sentry never even attempts to lookup symbols on my symbol server, which means that debug information is not properly populated.
I’ve tried a variety of different things like adding my symbol server to “SENTRY_BUILTIN_SOURCES” in the config file, but while it shows up on the UI and I can select it it still doesn’t work. I have also made sure that the symbolicator docker container can properly reach the symbol server, which it can. From the side of the symbol server I see no requests reaching it at all and on Sentry’s side it just says “Symbolication: missing”. Using the same symbol server in Visual Studio works as expected.
I should note a couple more things:
The reason I’m not using “upload_dif” to upload my PDBs directly to Sentry is because of storage considerations and because some of my PDBs are over the 2GB filesize limit.
My symbol server is running on a non-standard port (not port 80 / 443).
Sentry seems to be able to pull symbols from Microsoft’s public symbol server just fine.
I would greatly appreciate your help and please let me know if you require any additional information.
Hi @OrfeasZ and very sorry for the late reply. We have made some improvements and fixed some stuff with Sentry 10. Are you able to try again with the latest version and see if the issue persists?
Upon further investigation I managed to find the root of this issue. Our symbol server is accessible via a private IP in the 10.0.0.0/8 block. It appears that the symbolicator will block connection attempts to any private or reserved IP (debug message “Blocked attempt to connect to reserved IP address: X.X.X.X” is printed).
As a temporary workaround, I’ve built a custom version of the symbolicator that removes this check (src/utils/http.rs:111) and it can now pull symbols just fine. However, this is obviously not a real solution. Even though I’m not sure what the purpose of this restriction is, maybe there should be a configurable way to whitelist specific private addresses for people with similar configurations to mine?
@OrfeasZ - oh, thanks a lot for digging more into this and sorry for the late response! I’ll call in some Symbolicator folks to shed more light on this.
@OrfeasZ did you try to set this configuration in your config.yml (documented here):
connect_to_reserved_ips: true
This config value is intended to solve your issue, and please let me know if it doesn’t work. We would issue a bugfix in that case. We chose to set the default to false to be conservative as this may impose a security risk.
I am aware that it is hard to debug these sorts of issues. We’re already looking into improving observability, such as surfacing failed connections to symbol servers in Sentry.