Snuba-events-subscriptions-consumers is not working

[seungjinlee]

Currently I am running the event topic divided into 10 partitions. In all other cases, there is no problem, only snuba-events-subscriptions-consumers are constantly rebalancing or the current offset remains at zero. When I first set it up, I didn’t have this problem. At the time, snuba-events-subscriptions-consumers also worked well. However, once the problem occurs, initializing the offset with “–reset-offsets --to-latest --execute” does not solve the problem. You need a solid solution.

[BYK]

@markstory @fpacifici any ideas around this?

[fpacifici]

Hi, sorry for the late answer.

I would need some clarifications.

• Does snuba-transactions-subscriptions-consumers have the same problem? They run with the same consumer, they should behave in the same way.
• Do you have some consumer logs of the case where snuba-events-subscriptions-consumers keeps rebalancing? How often? Does it actually reshuffle partitions or does it reassign the same to the same consumers? The subscription consumer consumer from two topics: events and snuba-commit-log, so I would be interested in seeing how partitions are assigned there.
• Would you be able to start the subscription consumer in debug mode? -log-level DEBUG and show what is actually doing ?

Thanks
Filippo

[seungjinlee]

sorry, i almost forgot this thread!
At first glance, the service didn’t seem to have any direct damage, so the problem was neglected.
And, as you said, snuba-transactions-subscriptions-consumers also doesn’t work. These two topics are constantly rebalancing as soon as the consumer attaches. Consumers are restarted continuously.

this is out error log!

Caught KeyError(8), shutting down…
Traceback (most recent call last):
File “/usr/local/bin/snuba”, line 33, in
sys.exit(load_entry_point(‘snuba’, ‘console_scripts’, ‘snuba’)())
File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/usr/src/snuba/snuba/cli/subscriptions.py”, line 224, in subscriptions
batching_consumer.run()
File “/usr/src/snuba/snuba/utils/streams/processing/processor.py”, line 109, in run
self._run_once()
File “/usr/src/snuba/snuba/utils/streams/processing/processor.py”, line 147, in _run_once
self.__processing_strategy.submit(self.__message)
File “/usr/src/snuba/snuba/utils/streams/processing/strategies/batching.py”, line 152, in submit
result = self.__worker.process_message(message)
File “/usr/src/snuba/snuba/subscriptions/worker.py”, line 105, in process_message
for task in self.__schedulers[message.partition.index].find(tick.timestamps)
KeyError: 8