Sentry 10 events are not fully processed

On-Premise
1

I’m trying to piece together a kubernetes setup based on the docker-compose -based setup. Slow working, but I’m getting there.

At this point, everything seems to be working, except for the most important part: issues.
If I fire test events at my instance, I can see the count in the “stats” page move. However, the issue list remains empty.

Sentry v10 has a LOT of moving parts, so I have no idea what path events travel along as they get processed. I guess I’m asking for hints as to where I should start looking for errors.

2

oops, figured it out. My snuba-consumer needed a restart.

3

Heh, I was gonna suggest looking into post-process-forwarder service but you beat me to it.

4

Could you share specifically how to accomplish this? I’m running into a very similar (possibly the same) issue and like OP said there are a lot more moving parts in v10 that it’s hard to identify how to accomplish this fix.

5

Make sure you have this fix on your local copy: fix(post-process): Add missing post-process-forwarder by BYK · Pull Request #309 · getsentry/self-hosted · GitHub

And then make sure that new service is up and running :slight_smile:

6

I do have that line in my docker-compose.yml, and when I run docker-compose ps I believe it shows that it’s running (along with all the other services say they’re up):

image
image866×27 1.54 KB

I’ve tried restarting that service as well as snuba-consumer to no avail. :confused:

The output from sentry queues list returns with the relevant lines (all other lines show 0):

events.preprocess_event 3246
events.process_event 0
events.reprocess_events 0
events.reprocessing.preprocess_event 0
events.reprocessing.process_event 0
events.save_event 5

Is there any way to flush the queues somehow? Or am I not even looking at the right place for that?

7

I think this indicates a stuck worker. I’d try restarting the worker service and see what happens.

8

Okay I tried restarting that and nothing changed, but I ran docker-compose logs worker and I think I may have found part of what’s going wrong?

I think the most relevant part is:
index "pg_toast_21236_index" contains unexpected zero page at block 1836\nHINT: Please REINDEX it.

Assuming that’s the culprit, how would I go about fixing that?

Attaching to sentry_onpremise_worker_1
worker_1 | SQL: INSERT INTO "nodestore_node" ("id", "data", "timestamp") VALUES (%s, %s, %s)
worker_1 | 20:56:41 [ERROR] celery.worker.job: Task sentry.tasks.store.save_event[63b954f7-23e3-4e0a-aa44-b8cb87cdc99c] raised unexpected: InternalError(u'IndexCorrupted(\'index "pg_toast_21236_index" contains unexpected zero page at block 1836\\nHINT:  Please REINDEX it.\\n\',)\nSQL: INSERT INTO "nodestore_node" ("id", "data", "timestamp") VALUES (%s, %s, %s)',) (data={u'hostname': 'celery@78be3d5670cb', u'name': 'sentry.tasks.store.save_event', u'args': '[]', u'internal': False, u'kwargs': "{'event_id': u'935c49b3d68b4021a4ad9dd28de8fdab', 'cache_key': u'e:935c49b3d68b4021a4ad9dd28de8fdab:1', 'start_time': 1580330751.202655, 'project_id': 1L, 'data': None}", u'id': '63b954f7-23e3-4e0a-aa44-b8cb87cdc99c'})
worker_1 | Traceback (most recent call last):
worker_1 |   File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
worker_1 |     R = retval = fun(*args, **kwargs)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
worker_1 |     return self.run(*args, **kwargs)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/celery.py", line 161, in _inner
worker_1 |     reraise(*exc_info)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/celery.py", line 156, in _inner
worker_1 |     return f(*args, **kwargs)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry/tasks/base.py", line 48, in _wrapped
worker_1 |     result = func(*args, **kwargs)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry/tasks/store.py", line 660, in save_event
worker_1 |     _do_save_event(cache_key, data, start_time, event_id, project_id, **kwargs)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry/tasks/store.py", line 591, in _do_save_event
worker_1 |     event = manager.save(project_id, assume_normalized=True)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry/event_manager.py", line 672, in save
worker_1 |     event.data.save()
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry/db/models/fields/node.py", line 151, in save
worker_1 |     nodestore.set(self.id, to_write)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry/utils/services.py", line 104, in <lambda>
worker_1 |     context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry/utils/metrics.py", line 152, in inner
worker_1 |     return f(*args, **kwargs)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry/nodestore/django/backend.py", line 46, in set
worker_1 |     create_or_update(Node, id=id, values={"data": data, "timestamp": timezone.now()})
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry/db/models/query.py", line 94, in create_or_update
worker_1 |     return objects.create(**create_kwargs), True
worker_1 |   File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 394, in create
worker_1 |     obj.save(force_insert=True, using=self.db)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 808, in save
worker_1 |     force_update=force_update, update_fields=update_fields)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 838, in save_base
worker_1 |     updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 924, in _save_table
worker_1 |     result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 963, in _do_insert
worker_1 |     using=using, raw=raw)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
worker_1 |     return getattr(self.get_queryset(), name)(*args, **kwargs)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 1079, in _insert
worker_1 |     return query.get_compiler(using=using).execute_sql(return_id)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1112, in execute_sql
worker_1 |     cursor.execute(sql, params)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/django/__init__.py", line 472, in execute
worker_1 |     return real_execute(self, sql, params)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
worker_1 |     return self.cursor.execute(sql, params)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
worker_1 |     six.reraise(dj_exc_type, dj_exc_value, traceback)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
worker_1 |     return self.cursor.execute(sql, params)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 81, in inner
worker_1 |     raise_the_exception(self.db, e)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 79, in inner
worker_1 |     return func(self, *args, **kwargs)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 22, in inner
worker_1 |     return func(self, *args, **kwargs)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 99, in inner
worker_1 |     six.reraise(exc_info[0], exc_info[0](msg), exc_info[2])
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 95, in inner
worker_1 |     return func(self, sql, *args, **kwargs)
worker_1 |   File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 78, in execute
worker_1 |     return self.cursor.execute(sql, clean_bad_params(params))
worker_1 | InternalError: IndexCorrupted('index "pg_toast_21236_index" contains unexpected zero page at block 1836\nHINT:  Please REINDEX it.\n',)
worker_1 | SQL: INSERT INTO "nodestore_node" ("id", "data", "timestamp") VALUES (%s, %s, %s)

Edit: I tried reindexing the given index in the postgres container to no avail (it said relation "pg_toast_21236_index" does not exist). I just hadn’t used the full index name. I ran REINDEX INDEX pg_toast.pg_toast_21236_index and that worked to get rid of the error.

9

Okay sorry for all the messages! I think that last step above resolved the issue. It took a while to clear out the backlog, and I’m sure there were a whole bunch of issues that got dropped, but it looks like it’s now successfully saving events again.

1 Like