Clikchouse backup/restore

On-Premise
1

Hello all,

Anyone tried to backup/restore clikchouse DB? (Sentry 20.10.1)
We meet issue that after restoring clickhouse DB there is no events anymore.

events_issue
events_issue1901×240 35.8 KB

2

I don’t think you can get help without sharing the details of your backup/restore process.

3

im doing binary backup :
1)stoping clickhouse -
2)rsync’ing those two folders from local to remote server :
/var/lib/clickhouse/data
/var/lib/clickhouse/metadata

  1. importing /var/lib/clickhouse/data /var/lib/clickhouse/metadata in another Sentry clickhouse server

I never user clickhouse before but this should work (I get approvement from clickhouse support)

@BYK
we got the same behavior (with no events) when we migrated from 9.12 to 20.10.1 and as workaround you suggested to use another image:(SENTRY_IMAGE=us.gcr.io/sentryio/sentry:aa4b37cefca6f83567e7874ccb13612ade00f5b1)
after using this image all events was in place.

so at the moment we are trying to make clikhouse backup exactly for server which was migrated from9.12 to 20.10.1 with this “custom image” and restore on another server.

Is this two issues can be somehow related? I mean is there possibility that we got now “no events” after restoring because we used workaround with another image when migrated from 9.12 to 20.10.1?

Kristaps

4

Can you provide the error message and stacktrace? An error loading data doesn’t necessarily mean that there isn’t any data.

5

hey @caseyduquettesc and thanks for involving.

this is logs from web container when im trying to open any event:

i can see that data persist in the database but looks like it cant be retrieved by dashboard.
i tried all clickshouse dump/restore methods (from official documentation) but still no luck

Kristaps

6

I could be wrong, but that stacktrace appears to indicate that the path doesn’t exist in your filestore, not clickhouse. What are your filestore settings?

7

hey,

this is filestorage config:
snip_storage

from my understanding all events should be stored in clickhouse, im wrong?

Kristaps

8

Correct, but the stacktrace you provided is not from loading events. It’s trying to read a file from your filestore. Can you verify that /data/files/17842/32341/27feddd8a49e4255a89d1a1162db153d exists in your filestore?

9

For complete event details you need both Clickhouse and Postgres (node store). You’d also need the filestore as @caseyduquettesc mentioned for minidumps etc.

@untitaker may provide more info I think.

10

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.