I do get following error in our production system regularly and wonder what could be the reason for it. Any idea how I can analyze / fix this issue.
Sidenote: For clickhouse I use AWS EFS (network share) to store the data.
2021-07-03T02:31:55.570+02:00
2021.07.03 00:31:55.570569 [ 56 ] {} void DB::BackgroundProcessingPool::threadFunction(): Poco::Exception. Code: 1000, e.code() = 2, e.displayText() = File not found: /var/lib/clickhouse/data/system/metric_log/delete_tmp_202107_264997_264997_0/columns.txt, Stack trace (when copying this message, always include the lines below):
| Field | Value |
|---|---|
| @ingestionTime | 1625272319926 |
| @log | 474751378558:sentry-prod |
| @logStream | sentry_onpremise_clickhouse_1 |
| @message | 2021.07.03 00:31:55.570569 [ 56 ] {} void DB::BackgroundProcessingPool::threadFunction(): Poco::Exception. Code: 1000, e.code() = 2, e.displayText() = File not found: /var/lib/clickhouse/data/system/metric_log/delete_tmp_202107_264997_264997_0/columns.txt, Stack trace (when copying this message, always include the lines below): |
| @timestamp | 1625272315570 |
2021-07-03T02:31:55.570+02:00
- Poco::FileNotFoundException::FileNotFoundException(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int) @ 0x1053f3f0 in /usr/bin/clickhouse
| Field | Value |
|---|---|
| @ingestionTime | 1625272319926 |
| @log | 474751378558:sentry-prod |
| @logStream | sentry_onpremise_clickhouse_1 |
| @message | 0. Poco::FileNotFoundException::FileNotFoundException(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int) @ 0x1053f3f0 in /usr/bin/clickhouse |
| @timestamp | 1625272315570 |
2021-07-03T02:31:55.544+02:00
2021.07.03 00:31:55.544836 [ 56 ] {} system.metric_log: Cannot quickly remove directory /var/lib/clickhouse/data/system/metric_log/delete_tmp_202107_264997_264997_0 by removing files; fallback to recursive removal. Reason: Code: 458, e.displayText() = DB::ErrnoException: Cannot unlink file /var/lib/clickhouse/data/system/metric_log/delete_tmp_202107_264997_264997_0/partition.dat, errno: 2, strerror: No such file or directory (version 20.3.9.70 (official build))
| Field | Value |
|---|---|
| @ingestionTime | 1625272319926 |
| @log | 474751378558:sentry-prod |
| @logStream | sentry_onpremise_clickhouse_1 |
| @message | 2021.07.03 00:31:55.544836 [ 56 ] {} system.metric_log: Cannot quickly remove directory /var/lib/clickhouse/data/system/metric_log/delete_tmp_202107_264997_264997_0 by removing files; fallback to recursive removal. Reason: Code: 458, e.displayText() = DB::ErrnoException: Cannot unlink file /var/lib/clickhouse/data/system/metric_log/delete_tmp_202107_264997_264997_0/partition.dat, errno: 2, strerror: No such file or directory (version 20.3.9.70 (official build)) |
| @timestamp | 1625272315544 |