Worker failing with `sentry.errors.events: preprocess.failed.empty`

On-Premise
1

Sentry 8.12.0, using the official docker container,

everything seemed to be working fine, but then all of the sudden sentry started reporting the new issues in the dashboard, clients get 200 when reporting and I can see the reports come in, but the worker fails to process them, I’m not sure if it’s related or not, but things seemed to have gone wrong once I added custom fingerprint to one of our clients.

image
c3c11784-d2c9-11e6-8d71-9e3506ba6c86.png212×690 6.4 KB

you can see the big drop at the end, and it’s happening for all projects even some that haven’t been updated recently.

looking at the worker log I see constant,

06:02:07 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key='e:d817b41584824f3ea017a543c2e34bf0:8')
06:02:07 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key='e:69291860bf5649f497d194254a426dd1:8')
06:02:07 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key='e:9ccef557e05f48c0a41c4536fb0c4294:8')
06:02:07 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key='e:6886f587400e4c38802d37445f735b85:8')
06:02:07 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key='e:ef796b8ab129405f8e263fdff87d138d:8')
06:02:08 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key='e:115a5aedb8de4561bdfdd42b7ae0cf27:8')
06:02:08 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key='e:aad17301ad5c41ccb00b722554baca6d:8')
06:02:08 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key='e:781a99a3f2204cf89b4c68efba2b37db:8')
06:02:08 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key='e:9103660875544666bdc84ff80b926c5b:8')
06:02:08 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key='e:5046b33cd0b54d04b639e71e6e6cb4c2:8')
06:02:08 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key='e:51f2a1ef29e6407391461f2a44ce2073:8')
06:02:08 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key='e:5eb78c5498f04a20b9b9ee06772000cb:8')
06:02:08 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key='e:2e563ae0fdd54d878d4e518808d029fe:8')
06:02:09 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key='e:fa8cb44c724e4a518958b677929b10b6:8')
06:02:09 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key='e:28fd83a610c249c898264bbb0a73725f:8')

I would really appreciate any help.

2

This error explicitly means “we can’t find the event in cache, but the job exists in the queue.”

This can happen in a couple scenarios:

  • These jobs were taking longer than 1 hour to get pulled from the queue and they got TTLd away, which is expected behavior.
  • The cache was purged and these keys are literally just missing.
  • Some other misconfiguration where maybe the blobs are being written into one cache, then trying to be read from a different one.

Other than that, I’m unsure what could be wrong. Hopefully this points you in the right direction.

3

Thanks for the info, it seems like the main issue was our server was being overwhelmed by reports causing it to fall back, and many of them started to timeout, I added a quata, and it seems like thing are back on track

I was wondering is there a document/article/diagram that shows the lifetime of a report as far as different system components are concerned? I’ve tried looking for one, but I wasn’t able to find it (besides https://docs.sentry.io/server/queue/ which is pretty light on details)

Here is what I’ve gathered and guessed,:

  • Report is sent by client to dashboard node
  • Dashboard node stores the raw event in the cache you mentioned. (Is this cache the same as what is referred to as queue/broker)?
  • The report is picked up from the cache by the worker node and updated in Postgres. (What I’m confused about is why the worker can pick up the cache key but not the item? are they stored in different systems and only one of those systems has TTL?).
1 Like
4

I’m having this exact issue as well!

Could you please explain how you managed to fix it? Not sure how I would go about adding a quata

Thanks

5

I believe he means quota/rate limit.

6

For what it’s worth, we ran into the same issue and ended up cleaning the backlog with:

sentry queues purge events.preprocess_event

From one of the worker nodes.

Once that was done, all backed up events were lost (as expected) but processing resumed as normal and new events are treated properly.

7

Hello guys

I know that it’s an old post, but I just had the same error and the purge didn’t help me.

So here’s what I did:

I have a docker on premise configuration, where I had to add a special network (in order to see sentry in my nginx-proxy).
Using “sentry queues list” on the web instance and on the worker instance, I found out that both of them where not linked to the same redis instances.

So the solution for me was, edit your docker-compose.yml, and add links in your default conf like so:

x-defaults: &defaults
  restart: unless-stopped
  build: .
  links:
     - redis:redis
     - postgres:postgres
     - memcached:memcached
     - smtp:smtp

then restart your instances and problem should be fixed :wink: