Upgrade to Sentry 10 - Receving Events Error

On-Premise
1

Hi,

I recently upgraded sentry from V8 -> V9.1.2 -> V10. We are still using old sentry API’s (such as raven-python). I can see events coming in under the Stats tab, but under issues, no new events are shown. Taking a look at the logs in sentry_onpremise_snuba-consumer_1 I see this log:

Traceback (most recent call last):
  File "/usr/local/bin/snuba", line 11, in <module>
    load_entry_point('snuba', 'console_scripts', 'snuba')()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/src/snuba/snuba/cli/consumer.py", line 178, in consumer
    consumer.run()
  File "/usr/src/snuba/snuba/utils/streams/batching.py", line 137, in run
    self._run_once()
  File "/usr/src/snuba/snuba/utils/streams/batching.py", line 152, in _run_once
    self._handle_message(msg)
  File "/usr/src/snuba/snuba/utils/streams/batching.py", line 177, in _handle_message
    result = self.worker.process_message(msg)
  File "/usr/src/snuba/snuba/consumer.py", line 64, in process_message
    processed = self._process_message_impl(value, metadata)
  File "/usr/src/snuba/snuba/consumer.py", line 81, in _process_message_impl
    return processor.process_message(value, metadata)
  File "/usr/src/snuba/snuba/datasets/events_processor_base.py", line 150, in process_message
    processed = self.process_insert(event, metadata)
  File "/usr/src/snuba/snuba/datasets/events_processor_base.py", line 208, in process_insert
    datetime.strptime(event["datetime"], settings.PAYLOAD_DATETIME_FORMAT),
TypeError: strptime() argument 1 must be str, not None

Is this an error that was caused by the old API, or the migration? Thanks for your help.

2

Can you share the exact Sentry version you are using? Is it 10.0.0 or something newer?

3

Hi,

The version is: Sentry 10.1.0.dev 0a38e6ba

4

@codydietrich sorry for the late response. I wasn’t able to track down 0a38e6ba. Would you mind performing an upgrade and see if that fixes your issue?

5

@BYK Thanks for the update. We were able to do some manual debugging and found some of these events from Sentry 8 did not have the datetime field. We modified the code to set a default value, so the import of the old events worked.

However, now that we are up and running, it seems we run out of space (~145GB) every few days due to the Kafka volume growing to 55GB. Does this mean our events are overloading the system to where there are so many events backed up in Kafka?

EDIT:
Since I’m not familiar with Kafka, I assumed it was deleting the messages after they were consumed, which is not what Kafka does. To fix this issue, I set our retention period to 24 hours instead of the default of 7 days. Are there any downsides to this retention period? Does sentry need to go back and replay these messages?

1 Like
6

Hmm, wasn’t aware of this. I don’t think we need the whole 7 days of retention inside Kafka to be honest. Sentry normally doesn’t go back in time and replay messages except for a crash and restart. I think 24h is a much more reasonable time span. What do you think @matt @jauer and @untitaker?

7

It’s just up to your risk tolerances. If things crash and are down for more than 24h, you’ll have other issues to dig out of since the offset needed is missing. So on top of losing data, you’ll have to manually reset offsets unless you’re configured to always go to latest or something when it’s missing. There’s not really a right or wrong here.

1 Like
8

Thanks for the information, this is helpful in deciding what the appropriate retention period should be!