Uploading minidumps using Python

On-Premise
1

Hey,

I’m trying to upload minidump of size ~150MB using Python3:

    for dump in dumps:
        params = (
            ('sentry_key', '<key>'),
        )
        files = {
            'upload_file_minidump': open(dump, 'rb')
        }
        requests.post(SENTRY_UPLOAD_API, params=params, files=files)

But for some reason I’m getting ConnectionError:

Traceback (most recent call last):
File “D:\Projects\my_project\venv\lib\site-packages\urllib3\connectionpool.py”, line 603, in urlopen
chunked=chunked)
File “D:\Projects\my_project\venv\lib\site-packages\urllib3\connectionpool.py”, line 387, in _make_request
six.raise_from(e, None)
File “”, line 2, in raise_from
File “D:\Projects\my_project\venv\lib\site-packages\urllib3\connectionpool.py”, line 383, in _make_request
httplib_response = conn.getresponse()
File “D:\Program Files (x86)\Python\Python37-32\lib\http\client.py”, line 1344, in getresponse
response.begin()
File “D:\Program Files (x86)\Python\Python37-32\lib\http\client.py”, line 306, in begin
version, status, reason = self._read_status()
File “D:\Program Files (x86)\Python\Python37-32\lib\http\client.py”, line 275, in _read_status
raise RemoteDisconnected(“Remote end closed connection without”
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “D:\Projects\my_project\venv\lib\site-packages\requests\adapters.py”, line 449, in send
timeout=timeout
File “D:\Projects\my_project\venv\lib\site-packages\urllib3\connectionpool.py”, line 641, in urlopen
_stacktrace=sys.exc_info()[2])
File “D:\Projects\my_project\venv\lib\site-packages\urllib3\util\retry.py”, line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File “D:\Projects\my_project\venv\lib\site-packages\urllib3\packages\six.py”, line 685, in reraise
raise value.with_traceback(tb)
File “D:\Projects\my_project\venv\lib\site-packages\urllib3\connectionpool.py”, line 603, in urlopen
chunked=chunked)
File “D:\Projects\my_project\venv\lib\site-packages\urllib3\connectionpool.py”, line 387, in _make_request
six.raise_from(e, None)
File “”, line 2, in raise_from
File “D:\Projects\my_project\venv\lib\site-packages\urllib3\connectionpool.py”, line 383, in _make_request
httplib_response = conn.getresponse()
File “D:\Program Files (x86)\Python\Python37-32\lib\http\client.py”, line 1344, in getresponse
response.begin()
File “D:\Program Files (x86)\Python\Python37-32\lib\http\client.py”, line 306, in begin
version, status, reason = self._read_status()
File “D:\Program Files (x86)\Python\Python37-32\lib\http\client.py”, line 275, in _read_status
raise RemoteDisconnected(“Remote end closed connection without”
urllib3.exceptions.ProtocolError: (‘Connection aborted.’, RemoteDisconnected(‘Remote end closed connection without response’))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “D:\Projects\my_project\venv\lib\site-packages\requests\api.py”, line 60, in request
return session.request(method=method, url=url, **kwargs)
File “D:\Projects\my_project\venv\lib\site-packages\requests\sessions.py”, line 533, in request
resp = self.send(prep, **send_kwargs)
File “D:\Projects\my_project\venv\lib\site-packages\requests\sessions.py”, line 646, in send
r = adapter.send(request, **kwargs)
File “D:\Projects\my_project\venv\lib\site-packages\requests\adapters.py”, line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (‘Connection aborted.’, RemoteDisconnected(‘Remote end closed connection without response’))

I’ve spent half of a day googling, tried changing User-Agent as someone suggested that standard one might be the reason, but so far no luck. Can anyone help? Thanks in advance.

Sentry 10.1.0.dev 05e720a7

2

Hi,

Looks like you are running Sentry from Windows in a venv which is kind of shocking to me as it should not work (due to uwsgi and librdkafka not compiling on Windows), how did you manage to get it running? :slight_smile:

Regarding your issue, it maybe due to an earlier configuration error. Can you make sure this setting is in your settings file: https://github.com/getsentry/onpremise/commit/fd9bd107df57637f6901e14716ea62a63ba4cd3b

3

Haha, no, those exceptions come from my script (running on Windows) that downloads zip file with dumps, extracts them and sends to Sentry running on VM (guest OS: Ubuntu 18.04). I tried running the same script inside VM, but then I’m getting BrokenPipeError.

Indeed entry for chunked upload was missing, but there it didn’t make any difference. Do I need to run something, so Sentry would notice new entry in sentry.conf.py? I simply started it after making the change.

4

OK, so I found out in logs that by default size of file that Sentry accepts is limited:
Invalid (too big) CONTENT_LENGTH. skip.
hence empty response from server. I’ve setup new VM with Sentry, adding line

"limit-post": 1024 * 1024 * 1024

in sentry.conf.py beforehand. Thanks to this, I was able to upload 110 MB dump, but now it fails with 289 MB file. In logs there is traceback:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/sentry/web/api.py", line 426, in dispatch
    request, helper, project_config, origin=origin, *args, **kwargs
  File "/usr/local/lib/python2.7/site-packages/sentry/web/api.py", line 740, in _dispatch
    **kwargs
  File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py", line 88, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/web/api.py", line 903, in post
    **kwargs
  File "/usr/local/lib/python2.7/site-packages/sentry/web/api.py", line 652, in process
    event_manager, project, key, remote_addr, helper, attachments, project_config
  File "/usr/local/lib/python2.7/site-packages/sentry/web/api.py", line 291, in process_event
    helper.insert_data_to_database(data, start_time=start_time, attachments=attachments)
  File "/usr/local/lib/python2.7/site-packages/sentry/coreapi.py", line 160, in insert_data_to_database
    attachment_cache.set(cache_key, attachments, cache_timeout)
  File "/usr/local/lib/python2.7/site-packages/sentry/attachments/base.py", line 107, in set
    metrics_tags=metrics_tags,
  File "/usr/local/lib/python2.7/site-packages/sentry/attachments/base.py", line 128, in set_unchunked_data
    self.inner.set(key, compressed, timeout, raw=True)
  File "/usr/local/lib/python2.7/site-packages/sentry/cache/redis.py", line 25, in set
    raise ValueTooLarge("Cache key too large: %r %r" % (key, len(v)))
ValueTooLarge: Cache key too large: u'c:1:e:c71f06218e1f4eeabf1cfa8170f48077:2:a:0' 110239982
10:59:37 [ERROR] sentry: Cache key too large: u'c:1:e:c71f06218e1f4eeabf1cfa8170f48077:2:a:0' 110239982
192.168.1.3 - - [23/Feb/2020:10:55:10 +0000] "POST /api/2/minidump/?sentry_key=<my_key> HTTP/1.1" 500 633 "-" "curl/7.55.1"

It looks like default max size is 50 MB:

max_size = 50 * 1024 * 1024  # 50MB

What’s the best way to increase it and how big it should be? Not sure if it matters here, but my script downloads several dumps at once and then uploads them one after another.

5

Our resident minidump expert* shared the following with me:

since the limit is not configurable, there are only hacky workarounds for this, unfortunately. a few thoughts:

  • I would advise against uploading such large minidumps. this has severe impact on symbolication performance and unless larger portions of the heap are needed, i would recommend to configure the client to create smaller dumps.
  • In the setup file, it would probably be possible to patch the attribute on this class and increase the value to something higher
  • Unfortunately, the cache is required for processing, but it can be configured to the DjangoCache, in which case it might accept larger values (depending on what the backend for that one is)

* this is a made up title as you may have guessed, it was @jauer