Sentry 10 error after updating Docker images

On-Premise
1

We re-built our on-prem Sentry stack based on the newest “latest” image.
After upgrading we execute sentry upgrade in the web or worker container. I also attempted to “upgrade” snuba/clickhouse by running snuba bootstrap --force in the snubaapi container. However, I’m getting a bunch of errors and warnings:

2020-02-17 09:26:39,141 Column 'transaction_hash' type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(transaction_name), is: UInt64)
2020-02-17 09:26:39,141 Column 'transaction_status' type differs between local ClickHouse and schema! (expected: UInt8 DEFAULT 2, is: UInt8)
2020-02-17 09:26:39,141 Column '_start_date' type differs between local ClickHouse and schema! (expected: Date MATERIALIZED toDate(start_ts), is: Date)
2020-02-17 09:26:39,141 Column '_finish_date' type differs between local ClickHouse and schema! (expected: Date MATERIALIZED toDate(finish_ts), is: Date)
2020-02-17 09:26:39,141 Column 'user' type differs between local ClickHouse and schema! (expected: String DEFAULT '', is: String)
2020-02-17 09:26:39,141 Column 'user_hash' type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(user), is: UInt64)

(many of these)

I’m also getting this one:

clickhouse_driver.errors.ServerException: Code: 62.
DB::Exception: Syntax error: failed at position 2152 (line 7, col 13): TTL timestamp + toIntervalDay(retention_days)
            SETTINGS index_granularity=8192;. Expected one of: NOT, SETTINGS, FORMAT, LIKE, AND, OR, IN, BETWEEN, token, IS, NOT LIKE, NOT IN, GLOBAL IN, GLOBAL NOT IN, Comma, Dot, Arrow, QuestionMark, PRIMARY KEY, INTO OUTFILE. Stack trace:

So I guess my question is:

  1. How do I sync clickhouse and Schema? (I thought that was what I did when running snuba bootstrap)
  2. How can I address the syntax error exception?
2

Running the install script should take care of this. It runs snuba migrate on https://github.com/getsentry/onpremise/blob/9dfc5c99defd01b7978a5988657581e7c668461b/install.sh#L117 which is the command you are looking for. snuba bootstrap is only when you are starting from scratch.

This, I don’t know. I’d update both snuba and Clickhouse. The version we use is now 19.11: https://github.com/getsentry/onpremise/blob/9dfc5c99defd01b7978a5988657581e7c668461b/docker-compose.yml#L99

3

Thanks!!

4

That helped a lot. However, I’m getting a different error:
DB::Exception: Table default.errors_local doesn't exist
Anyone know how to fix that one?

5

Fixed it. Running snuba bootstrap --force folowed by the migrate command seems to have fixed it. I’m still getting some stuff like 'event_id' type differs between local ClickHouse and schema! but everything seems to work so I guess we can live with that.

1 Like
6

Running another migrate should solve that I think: docker-compose run --rm snuba-api migrate :slight_smile:

7

I’m also getting messages about type differences each time I run the install.sh script. Here are some of these messages:

2020-03-03 12:37:38,003 Migrating dataset transactions
2020-03-03 12:37:38,006 Column 'transaction_hash' type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(transaction_name), is: UInt64)
2020-03-03 12:37:38,006 Column 'transaction_status' type differs between local ClickHouse and schema! (expected: UInt8 DEFAULT 2, is: UInt8)
2020-03-03 12:37:38,006 Column '_start_date' type differs between local ClickHouse and schema! (expected: Date MATERIALIZED toDate(start_ts), is: Date)
2020-03-03 12:37:38,006 Column '_finish_date' type differs between local ClickHouse and schema! (expected: Date MATERIALIZED toDate(finish_ts), is: Date)
2020-03-03 12:37:38,006 Column 'user' type differs between local ClickHouse and schema! (expected: String DEFAULT '', is: String)
2020-03-03 12:37:38,006 Column 'user_hash' type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(user), is: UInt64)
2020-03-03 12:37:38,006 Column 'sdk_name' type differs between local ClickHouse and schema! (expected: LowCardinality(String) DEFAULT '', is: LowCardinality(String))
2020-03-03 12:37:38,006 Column 'sdk_version' type differs between local ClickHouse and schema! (expected: LowCardinality(String) DEFAULT '', is: LowCardinality(String))
2020-03-03 12:37:38,009 Column 'transaction_hash' type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(transaction_name), is: UInt64)
2020-03-03 12:37:38,009 Column 'transaction_status' type differs between local ClickHouse and schema! (expected: UInt8 DEFAULT 2, is: UInt8)
2020-03-03 12:37:38,009 Column '_start_date' type differs between local ClickHouse and schema! (expected: Date MATERIALIZED toDate(start_ts), is: Date)
2020-03-03 12:37:38,009 Column '_finish_date' type differs between local ClickHouse and schema! (expected: Date MATERIALIZED toDate(finish_ts), is: Date)
2020-03-03 12:37:38,009 Column 'user' type differs between local ClickHouse and schema! (expected: String DEFAULT '', is: String)
2020-03-03 12:37:38,009 Column 'user_hash' type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(user), is: UInt64)
2020-03-03 12:37:38,009 Column 'sdk_name' type differs between local ClickHouse and schema! (expected: LowCardinality(String) DEFAULT '', is: LowCardinality(String))
2020-03-03 12:37:38,009 Column 'sdk_version' type differs between local ClickHouse and schema! (expected: LowCardinality(String) DEFAULT '', is: LowCardinality(String))
2020-03-03 12:37:38,010 Migrating dataset events_migration
2020-03-03 12:37:38,013 Column 'event_id' type differs between local ClickHouse and schema! (expected: UUID CODEC (NONE), is: UUID)
2020-03-03 12:37:38,013 Column 'event_hash' type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(toString(event_id)) CODEC (NONE), is: UInt64)
2020-03-03 12:37:38,014 Column 'user' type differs between local ClickHouse and schema! (expected: String DEFAULT '', is: String)
2020-03-03 12:37:38,014 Column 'user_hash' type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(user), is: UInt64)
2020-03-03 12:37:38,014 Column 'transaction_name' type differs between local ClickHouse and schema! (expected: LowCardinality(String) DEFAULT '', is: LowCardinality(String))
2020-03-03 12:37:38,014 Column 'transaction_hash' type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(transaction_name), is: UInt64)
2020-03-03 12:37:38,014 Column 'offset' type differs between local ClickHouse and schema! (expected: UInt64 CODEC (DoubleDelta, LZ4), is: UInt64)
2020-03-03 12:37:38,014 Column 'primary_hash_hex' type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED hex(primary_hash), is: UInt64)
2020-03-03 12:37:38,014 Column 'event_string' type differs between local ClickHouse and schema! (expected: String CODEC (NONE), is: String)
2020-03-03 12:37:38,017 Column 'event_id' type differs between local ClickHouse and schema! (expected: UUID CODEC (NONE), is: UUID)
2020-03-03 12:37:38,017 Column 'event_hash' type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(toString(event_id)) CODEC (NONE), is: UInt64)
2020-03-03 12:37:38,017 Column 'user' type differs between local ClickHouse and schema! (expected: String DEFAULT '', is: String)
2020-03-03 12:37:38,017 Column 'user_hash' type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(user), is: UInt64)
2020-03-03 12:37:38,017 Column 'transaction_name' type differs between local ClickHouse and schema! (expected: LowCardinality(String) DEFAULT '', is: LowCardinality(String))
2020-03-03 12:37:38,017 Column 'transaction_hash' type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED cityHash64(transaction_name), is: UInt64)
2020-03-03 12:37:38,017 Column 'offset' type differs between local ClickHouse and schema! (expected: UInt64 CODEC (DoubleDelta, LZ4), is: UInt64)
2020-03-03 12:37:38,017 Column 'primary_hash_hex' type differs between local ClickHouse and schema! (expected: UInt64 MATERIALIZED hex(primary_hash), is: UInt64)
2020-03-03 12:37:38,017 Column 'event_string' type differs between local ClickHouse and schema! (expected: String CODEC (NONE), is: String)
2020-03-03 12:37:38,018 Migrating dataset outcomes
2020-03-03 12:37:38,020 Column 'size' exists in local ClickHouse but not in schema!

Running docker-compose run --rm snuba-api migrate doesn’t help unfortunately.

Can I safely ignore those messages?

Thanks a lot for your help!

8

Getting the same errors on a clean installation.

9

This should not be an issue with the most recent versions. @MoshiBin are you sure you have the latest version of the onpremise repo and the docker images? The install script already runs docker-compose pull to get them but asking just in case.

@lextar - the migrate command should actually take care of this. Are you still getting them? Can you also update your local version of onpremise from our version as we bumped the version of Clickhouse.