Snuba consumer crashes during migration 9.1.2 -> 21.x

On-Premise
1

During the upgrade from 9.1.2 the snuba consumer pod on our Kubernetes Deployment (bare metal, converted docker-compose.yaml via ‘kompose’) keeps crashing after a few hours during the upgrade process:

 2021-02-06 14:34:46,616 New partitions assigned:     {Partition(topic=Topic(name='events'), index=0): 163127}                                                                                                         
 2021-02-06 14:34:46,863 Caught KeyError('received'), shutting down...                                                                                                                                             
 Traceback (most recent call last):                                                                                                                                                                                
   File "/usr/local/bin/snuba", line 33, in <module>                                                                                                                                                               
     sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())                                                                                                                                             
   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__                                                                                                                              
     return self.main(*args, **kwargs)                                                                                                                                                                             
   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main                                                                                                                                  
     rv = self.invoke(ctx)                                                                                                                                                                                         
   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke                                                                                                                               
     return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                       
   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke                                                                                                                               
     return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                
   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke                                                                                                                                
     return callback(*args, **kwargs)                                                                                                                                                                              
   File "/usr/src/snuba/snuba/cli/consumer.py", line 161, in consumer                                                                                                                                              
     consumer.run()                                                                                                                                                                                                
   File "/usr/src/snuba/snuba/utils/streams/processing/processor.py", line 109, in run                                                                                                                             
     self._run_once()                                                                                                                                                                                              
   File "/usr/src/snuba/snuba/utils/streams/processing/processor.py", line 147, in _run_once                                                                                                                       
     self.__processing_strategy.submit(self.__message)                                                                                                                                                             
   File "/usr/src/snuba/snuba/utils/streams/processing/strategies/streaming/transform.py", line 64, in submit                                                                                                      
     self.__transform_function(message),                                                                                                                                                                           
   File "/usr/src/snuba/snuba/consumer.py", line 253, in process_message                                                                                                                                           
     result = processor.process_message(                                                                                                                                                                           
   File "/usr/src/snuba/snuba/datasets/events_processor_base.py", line 154, in process_message                                                                                                                     
     row = self.process_insert(event, metadata)                                                                                                                                                                    
   File "/usr/src/snuba/snuba/datasets/events_processor_base.py", line 189, in process_insert                                                                                                                      
     self.extract_common(processed, event, metadata)                                                                                                                                                               
   File "/usr/src/snuba/snuba/datasets/events_processor_base.py", line 230, in extract_common                                                                                                                      
     received = _collapse_uint32(int(data["received"]))                                                                                                                                                            
 KeyError: 'received'

The upgrade finished after a few hours - so far succesfully, but the snuba consumer pod does’nt start up anymore - keeps crashing with the same error.

Sentry Version we are using is:
sentry, version 21.2.0.dev0 (718fa19bc161)
(had an issue with an unhandled exception during the upgrade - Error importing events during upgrade from 9.1.2 to 20.11 (and up) - #9 by BYK)

2

@lynnagara any ideas?

3

just getting a little more specific: this only happens during upgrading with importing the events.
I don’t get this error with a clean install or when skipping importing all the issues/events with: SENTRY_SKIP_EVENTS_BACKFILL_FOR_10=1

4

Hi @sraisl

If I had to guess, it sounds like you might have at least one event that isn’t valid - specifically missing both the “received” or “timestamp” value in the events and nodestore tables in postgres, which is where the script looks for the received value. Are you able to check whether that might be the case?

5

Hi @lynnagara!
Sure. Could you tell me which tables to check in particular?
I checked for “received” or “timestamp” columns in “nodestore_node” and “sentry_eventmapping” but these tables don’t have these columns…

6

Hi @sraisl, apologies just double checked the table names. The script looks in 2 potential places for a received value:

  1. nodestore_node table - in the “data” gzipped dict column, a value for the “received” key
  2. sentry_message table - the “datetime” column (this table only exists before the upgrade, it should have been dropped afterwards)
1 Like
7

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.