Support for Clickhouse 21.x

I am deploying Sentry on Openshift and due to how it works I have to use Clickhouse >= 21.3.2.5 (Clickhouse developers introduced a few changes to make it easier to run the server on Openshift). This seems to be causing problems for snuba api. Does Sentry support one of the recent versions of Clickhouse? For example snuba-outcomes-consumer exits with the following error:

2021-03-19 08:45:22,781 New partitions assigned: {Partition(topic=Topic(name='outcomes'), index=0): 6}
2021-03-19 08:45:22,793 Caught HTTPError('Received unexpected 400 response: '), shutting down...
Traceback (most recent call last):
  File "/usr/local/bin/snuba", line 33, in <module>
    sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/src/snuba/snuba/cli/consumer.py", line 161, in consumer
    consumer.run()
  File "/usr/src/snuba/snuba/utils/streams/processing/processor.py", line 112, in run
    self._run_once()
  File "/usr/src/snuba/snuba/utils/streams/processing/processor.py", line 147, in _run_once
    self.__processing_strategy.poll()
  File "/usr/src/snuba/snuba/utils/streams/processing/strategies/streaming/transform.py", line 61, in poll
    self.__next_step.poll()
  File "/usr/src/snuba/snuba/utils/streams/processing/strategies/streaming/collect.py", line 119, in poll
    self.__close_and_reset_batch()
  File "/usr/src/snuba/snuba/utils/streams/processing/strategies/streaming/collect.py", line 104, in __close_and_reset_batch
    self.__batch.close()
  File "/usr/src/snuba/snuba/utils/streams/processing/strategies/streaming/collect.py", line 64, in close
    self.__step.close()
  File "/usr/src/snuba/snuba/consumers/consumer.py", line 220, in close
    self.__insert_batch_writer.close()
  File "/usr/src/snuba/snuba/consumers/consumer.py", line 91, in close
    self.__writer.write(
  File "/usr/src/snuba/snuba/clickhouse/http.py", line 180, in write
    batch.join()
  File "/usr/src/snuba/snuba/clickhouse/http.py", line 133, in join
    raise HTTPError(
urllib3.exceptions.HTTPError: Received unexpected 400 response: 

Errors happen e.g. when creating sample events in a project. /api/0/issues/3/events/latest/ returns 404. /api/0/organizations/sentry/issues/ returns 500.

More stacktraces:

Traceback (most recent call last):                                                                                                                                                                                                       
  File "/usr/local/lib/python3.6/site-packages/sentry/api/base.py", line 121, in handle_exception                                                                                                                                        
    response = super().handle_exception(exc)                                                                                                                                                                                             
  File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 449, in handle_exception                                                                                                                                   
    self.raise_uncaught_exception(exc)                                                                                                                                                                                                   
  File "/usr/local/lib/python3.6/site-packages/sentry/api/base.py", line 234, in dispatch                                                                                                                                                
    response = handler(request, *args, **kwargs)                                                                                                                                                                                         
  File "/usr/local/lib/python3.6/site-packages/sentry/api/helpers/group_index.py", line 487, in wrapper                                                                                                                                  
    return function(self, request, *args, **kwargs)                                                                                                                                                                                      
  File "/usr/local/lib/python3.6/site-packages/sentry/api/endpoints/organization_group_index.py", line 304, in get                                                                                                                       
    {"count_hits": True, "date_to": end, "date_from": start},                                                                                                                                                                            
  File "/usr/local/lib/python3.6/site-packages/sentry/api/endpoints/organization_group_index.py", line 165, in _search                                                                                                                   
    result = search.query(**query_kwargs)                                                                                                                                                                                                
  File "/usr/local/lib/python3.6/site-packages/sentry/search/snuba/backend.py", line 348, in query                                                                                                                                       
    max_hits=max_hits,                                                                                                                                                                                                                   
  File "/usr/local/lib/python3.6/site-packages/sentry/search/snuba/executors.py", line 472, in query                                                                                                                                     
    search_filters=search_filters,                                                                                                                                                                                                       
  File "/usr/local/lib/python3.6/site-packages/sentry/search/snuba/executors.py", line 200, in snuba_search                                                                                                                              
    condition_resolver=snuba.get_snuba_column_name,                                                                                                                                                                                      
  File "/usr/local/lib/python3.6/site-packages/sentry/utils/snuba.py", line 909, in aliased_query                                                                                                                                        
    return _aliased_query_impl(**kwargs)                                                                                                                                                                                                 
  File "/usr/local/lib/python3.6/site-packages/sentry/utils/snuba.py", line 977, in _aliased_query_impl                                                                                                                                  
    **kwargs,                                                                                                                                                                                                                            
  File "/usr/local/lib/python3.6/site-packages/sentry/utils/snuba.py", line 636, in raw_query                                                                                                                                            
    return bulk_raw_query([snuba_params], referrer=referrer)[0]                                                                                                                                                                          
  File "/usr/local/lib/python3.6/site-packages/sentry/utils/snuba.py", line 711, in bulk_raw_query                                                                                                                                       
    error["message"]                                                                                                                                                                                                                     
sentry.utils.snuba.QueryExecutionError: [27] DB::Exception: Cannot parse input: expected 'eof' before: 'rue'. Stack trace:                                                                                                               
                                                                                                                                                                                                                                         
0. DB::throwAtAssertionFailed(char const*, DB::ReadBuffer&) @ 0x86660a5 in /usr/bin/clickhouse                                                                                                                                           
1. ? @ 0xe77f4d1 in /usr/bin/clickhouse                                                                                                                                                                                                  
2. DB::BaseSettings<DB::SettingsTraits>::read(DB::ReadBuffer&, DB::SettingsWriteFormat) @ 0xe72c92d in /usr/bin/clickhouse                                                                                                               
3. DB::TCPHandler::receiveQuery() @ 0xf8c7b17 in /usr/bin/clickhouse                                                                                                                                                                     
4. DB::TCPHandler::receivePacket() @ 0xf8bf017 in /usr/bin/clickhouse                                                                                                                                                                    
5. DB::TCPHandler::runImpl() @ 0xf8b7812 in /usr/bin/clickhouse                                                                                                                                                                          
6. DB::TCPHandler::run() @ 0xf8ca1c9 in /usr/bin/clickhouse                                                                                                                                                                              
7. Poco::Net::TCPServerConnection::start() @ 0x11f7ccbf in /usr/bin/clickhouse                                                                                                                                                           
8. Poco::Net::TCPServerDispatcher::run() @ 0x11f7e6d1 in /usr/bin/clickhouse                                                                                                                                                             
9. Poco::PooledThread::run() @ 0x120b4df9 in /usr/bin/clickhouse                                                                                                                                                                         
10. Poco::ThreadImpl::runnableEntry(void*) @ 0x120b0c5a in /usr/bin/clickhouse                                                                                                                                                           
11. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so                                                                                                                                                                
12. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so                    

Kafka (2.7.0) logs:

625cdc81db with group instance id None) (kafka.coordinator.group.GroupCoordinator)
[2021-03-19 09:04:07,366] INFO [GroupCoordinator 2]: Stabilized group ingest-consumer generation 19 (__consumer_offsets-9) (kafka.coordinator.group.GroupCoordinator)
[2021-03-19 09:04:07,368] INFO [GroupCoordinator 2]: Assignment received from leader for group ingest-consumer for generation 19 (kafka.coordinator.group.GroupCoordinator)
[2021-03-19 09:04:17,372] INFO [GroupCoordinator 2]: Member rdkafka-2bbe2140-63ea-4bfd-b649-d5625cdc81db in group ingest-consumer has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2021-03-19 09:04:17,373] INFO [GroupCoordinator 2]: Preparing to rebalance group ingest-consumer in state PreparingRebalance with old generation 19 (__consumer_offsets-9) (reason: removing member rdkafka-2bbe2140-63ea-4bfd-b649-d5625cdc81db on heartbeat expiration) (kafka.coordinator.group.GroupCoordinator)

Using ClickHouse == 20.8.14.4 works fine but unfortunately I can’t use that version.

Hi @crossner90,

Yeah, as far as I know we are on Clickhouse 20 for now. Pinging @lynnagara and @fpacifici as they are the subject matter experts here.

Hi,

sorry for the late answer.

Unfortunately Clickhouse 21 is not supported by snuba at this time. We never even tested it on those versions, so it is hard to say what is supposed to work and what is not. It may even be just fine.

I did a quick test to verify your issue but I ran into different problems when creating the tables. Did you have any issue around that.
On the query side, there seem to be several breaking changes between what we use and the version you are using, I see some queries failing but none with failures when parsing the query string as you experience.
I’ll get back to you after doing some more tests, but it seems there will be multiple issues to fix to support 21.3.2.5

Filippo