Additional data is getting agressively truncated / trimmed

1

I’m sending some extra data with my Sentry error reports, and a lot of my data is going missing. It looks like it’s choosing some random path through the nested keys, and removing the rest of the keys in the parent objects.

For example, this:

{ 
  extra: { 
    state: { 
      store1: { 
        foo: {
          a: 1,
          b: 2,
        },
        bar: {
          ...
        },
        baz: {
          ...
        }
      },
      store2: {
        ...
      }
    }
  }
}

is being trimmed into this (removing the other keys):

{ 
  extra: { 
    state: { 
      store1: { 
        foo: {
          a: 1,
          b: 2,
        },
      },
    }
  }
}

I tried turning off “Data Scrubber” and “Use Default Scrubbers”, but that didn’t change anything.

Would this problem go away if I was using a self-hosted Sentry instead of sentry.io?

2

Hi @ndbroadbent, which SDK/language is this? I’m not aware that either sentry.io or the on-prem version would strip data like that, but the behavior within SDKs may vary.

3

Hi @untitaker, this is from raven-js in the browser, but I’m actually seeing the correct data in the POST request when I look at the developer tools. So it’s sending the correct data to Sentry, but when I look at the error report in the sentry UI, some of the data has gone missing.

If you are able to take a look, one example is event f5447ade3dfb40a199c07e2d29cf5094. And it’s the “state” key, in the Additional Data section.

EDIT: Another example is event e8cd2a8397284189afd326d4d1e1ec52. This time, it’s chosen to keep some different keys under “extra/state/editorStore”, and is discarding the rest of the data. But I can’t figure out what it’s doing. My first thought was I think I have some signed Amazon S3 URLs in the state, so they include credentials. But it didn’t actually scrub all of those URLs from the additional data, and turning off the “Data Scrubber” option didn’t help.

4

Hmm, I am not sure what keys are missing there. Could you give me an event payload (from the developer console) where you see sentry.io stripping data?

5

Sure, I’ve created a gist with the POST data copied from the developer console, and also the data that shows up in Sentry’s additional data: https://gist.github.com/ndbroadbent/d4a797f2cff2e0d7d21748ab58174805

This is for event 587657e028624dfa89f043a1f47ac1a7

The POST data is 76476 bytes, which is under the 200KB limit. But maybe there’s another limit that isn’t documented.

Also I can try stripping out some of the duplicate data in my state. I’m using Immutable.js records that have default values, so I can remove a lot of the nested keys if they are default values. Maybe that will shrink my data enough so that it isn’t trimmed.

6

Ah, I can reproduce this if I either run sentry locally, or run the following from sentry shell:

from sentry.utils.safe import trim_dict
trim_dict(event["extra"])

I’ll check if I can find somebody who knows this stuff better than me. I know that we are planning to change our pricing to fit larger event payloads but idk how it’s going to look like.

7

Oh ok, well it’s great to know that this is the expected behavior.

I’m actually on the free plan at the moment, so I would probably upgrade to a paid plan if there was a way to skip the trim_dict operation and store the full payload, because it’s incredibly useful to copy/paste the state into my local development server and replay the events to see what caused the crash. Or I might even migrate to a self-hosted version of Sentry and turn this off.

Thanks very much for your help!