Clickhouse DB::Exception: Table default.migrations_local doesn't exist

On-Premise
1

What is causing this error with clickhouse ?

clickhouse_1                                | Processing configuration file '/etc/clickhouse-server/config.xml'.
clickhouse_1                                | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
clickhouse_1                                | Merging configuration file '/etc/clickhouse-server/config.d/sentry.xml'.
clickhouse_1                                | Include not found: clickhouse_remote_servers
clickhouse_1                                | Include not found: clickhouse_compression
clickhouse_1                                | Logging information to /var/log/clickhouse-server/clickhouse-server.log
clickhouse_1                                | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
clickhouse_1                                | Logging information to console
clickhouse_1                                | 2021.03.02 13:03:04.109591 [ 1 ] {} <Information> : Starting ClickHouse 20.3.9.70 with revision 54433
clickhouse_1                                | 2021.03.02 13:03:04.111465 [ 1 ] {} <Information> Application: starting up
clickhouse_1                                | Include not found: networks
clickhouse_1                                | 2021.03.02 13:03:04.123642 [ 1 ] {} <Information> Application: Uncompressed cache size was lowered to 3.89 GiB because the system has low amount of memory
clickhouse_1                                | 2021.03.02 13:03:04.123960 [ 1 ] {} <Information> Application: Mark cache size was lowered to 3.89 GiB because the system has low amount of memory
clickhouse_1                                | 2021.03.02 13:03:04.123988 [ 1 ] {} <Information> Application: Loading metadata from /var/lib/clickhouse/
clickhouse_1                                | 2021.03.02 13:03:04.127204 [ 1 ] {} <Information> DatabaseOrdinary (default): Total 0 tables and 0 dictionaries.
clickhouse_1                                | 2021.03.02 13:03:04.127234 [ 1 ] {} <Information> DatabaseOrdinary (default): Starting up tables.
clickhouse_1                                | 2021.03.02 13:03:04.127662 [ 1 ] {} <Information> BackgroundSchedulePool: Create BackgroundSchedulePool with 16 threads
clickhouse_1                                | 2021.03.02 13:03:04.129415 [ 1 ] {} <Information> Application: It looks like the process has no CAP_NET_ADMIN capability, 'taskstats' performance statistics will be disabled. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_net_admin=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems. It also doesn't work if you run clickhouse-server inside network namespace as it happens in some containers.
clickhouse_1                                | 2021.03.02 13:03:04.129448 [ 1 ] {} <Information> Application: It looks like the process has no CAP_SYS_NICE capability, the setting 'os_thread_nice' will have no effect. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_sys_nice=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems.
clickhouse_1                                | 2021.03.02 13:03:04.133263 [ 1 ] {} <Error> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
clickhouse_1                                | 2021.03.02 13:03:04.133549 [ 1 ] {} <Error> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
clickhouse_1                                | 2021.03.02 13:03:04.133749 [ 1 ] {} <Error> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
clickhouse_1                                | 2021.03.02 13:03:04.133954 [ 1 ] {} <Error> Application: Listen [::]:9004 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
clickhouse_1                                | 2021.03.02 13:03:04.134040 [ 1 ] {} <Information> Application: Listening for http://0.0.0.0:8123
clickhouse_1                                | 2021.03.02 13:03:04.134079 [ 1 ] {} <Information> Application: Listening for connections with native protocol (tcp): 0.0.0.0:9000
clickhouse_1                                | 2021.03.02 13:03:04.134109 [ 1 ] {} <Information> Application: Listening for replica communication (interserver): http://0.0.0.0:9009
clickhouse_1                                | 2021.03.02 13:03:04.216828 [ 1 ] {} <Information> Application: Listening for MySQL compatibility protocol: 0.0.0.0:9004
clickhouse_1                                | 2021.03.02 13:03:04.218243 [ 1 ] {} <Information> Application: Available RAM: 7.79 GiB; physical cores: 4; logical cores: 4.
clickhouse_1                                | 2021.03.02 13:03:04.218276 [ 1 ] {} <Information> Application: Ready for connections.
clickhouse_1                                | Include not found: clickhouse_remote_servers
clickhouse_1                                | Include not found: clickhouse_compression
clickhouse_1                                | 2021.03.02 13:03:11.629659 [ 49 ] {} <Information> BackgroundProcessingPool: Create BackgroundProcessingPool with 16 threads
clickhouse_1                                | 2021.03.02 13:03:26.447907 [ 70 ] {46208108-980e-4910-a0f1-3bb61ed12df0} <Information> executeQuery: Read 1 rows, 1.00 B in 0.003 sec., 345 rows/sec., 345.13 B/sec.
clickhouse_1                                | 2021.03.02 13:03:26.448519 [ 70 ] {} <Information> TCPHandler: Processed in 0.004 sec.
clickhouse_1                                | 2021.03.02 13:03:26.494919 [ 70 ] {36df2ced-709e-40ae-87e1-ae885acd5222} <Error> executeQuery: Code: 60, e.displayText() = DB::Exception: Table default.migrations_local doesn't exist. (version 20.3.9.70 (official build)) (from 192.168.176.6:44902) (in query: SELECT group, migration_id, status FROM migrations_local FINAL WHERE group IN ('system', 'events', 'transactions', 'discover', 'outcomes', 'sessions')), Stack trace (when copying this message, always include the lines below):
clickhouse_1                                | 
clickhouse_1                                | 0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x105351b0 in /usr/bin/clickhouse
clickhouse_1                                | 1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x8f4172d in /usr/bin/clickhouse
clickhouse_1                                | 2. DB::Context::getTableImpl(DB::StorageID const&, std::__1::optional<DB::Exception>*) const @ 0xcfe2a24 in /usr/bin/clickhouse
clickhouse_1                                | 3. DB::Context::getTable(DB::StorageID const&) const @ 0xcfe2bbb in /usr/bin/clickhouse
clickhouse_1                                | 4. DB::Context::getTable(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0xcfe2c7d in /usr/bin/clickhouse
clickhouse_1                                | 5. DB::JoinedTables::getLeftTableStorage() @ 0xd454892 in /usr/bin/clickhouse
clickhouse_1                                | 6. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, std::__1::shared_ptr<DB::IBlockInputStream> const&, std::__1::optional<DB::Pipe>, std::__1::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xd13b6d1 in /usr/bin/clickhouse
clickhouse_1                                | 7. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xd13c619 in /usr/bin/clickhouse
clickhouse_1                                | 8. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xd341686 in /usr/bin/clickhouse
clickhouse_1                                | 9. DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, DB::Context&, DB::QueryProcessingStage::Enum) @ 0xd0909b4 in /usr/bin/clickhouse
clickhouse_1                                | 10. ? @ 0xd550655 in /usr/bin/clickhouse
clickhouse_1                                | 11. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool, bool) @ 0xd553441 in /usr/bin/clickhouse
clickhouse_1                                | 12. DB::TCPHandler::runImpl() @ 0x9024489 in /usr/bin/clickhouse
clickhouse_1                                | 13. DB::TCPHandler::run() @ 0x9025470 in /usr/bin/clickhouse
clickhouse_1                                | 14. Poco::Net::TCPServerConnection::start() @ 0xe3ac69b in /usr/bin/clickhouse
clickhouse_1                                | 15. Poco::Net::TCPServerDispatcher::run() @ 0xe3acb1d in /usr/bin/clickhouse
clickhouse_1                                | 16. Poco::PooledThread::run() @ 0x105c3317 in /usr/bin/clickhouse
clickhouse_1                                | 17. Poco::ThreadImpl::runnableEntry(void*) @ 0x105bf11c in /usr/bin/clickhouse
clickhouse_1                                | 18. ? @ 0x105c0abd in /usr/bin/clickhouse
clickhouse_1                                | 19. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
clickhouse_1                                | 20. __clone @ 0x12188f in /lib/x86_64-linux-gnu/libc-2.27.so
clickhouse_1                                | 
clickhouse_1                                | 2021.03.02 13:03:26.495729 [ 70 ] {} <Information> TCPHandler: Processed in 0.046 sec.
clickhouse_1                                | 2021.03.02 13:03:26.496149 [ 70 ] {} <Information> TCPHandler: Done processing connection.
clickhouse_1                                | 2021.03.02 13:03:26.511698 [ 70 ] {} <Information> TCPHandler: Processed in 0.005 sec.
clickhouse_1                                | 2021.03.02 13:03:26.514026 [ 70 ] {} <Information> TCPHandler: Processed in 0.001 sec.
clickhouse_1                                | 2021.03.02 13:03:26.621483 [ 70 ] {} <Information> TCPHandler: Processed in 0.107 sec.
clickhouse_1                                | 2021.03.02 13:03:26.629242 [ 70 ] {} <Information> TCPHandler: Processed in 0.003 sec.
clickhouse_1                                | 2021.03.02 13:03:26.636906 [ 70 ] {} <Information> TCPHandler: Processed in 0.007 sec.
clickhouse_1                                | 2021.03.02 13:03:26.648516 [ 70 ] {} <Information> TCPHandler: Processed in 0.010 sec.
2

Looks like you haven’t run snuba bootstrap?

3

That is not possible. I have this on the first run of the components, i.e right after ./install.sh and doing docker-compose up -d.
I am not doing anything else there. The only think I can think of is the snuba bootstrap is not ran completely when the clickhouse being initialized.

4

It is also strange that I have

<Error> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>

in spite of, I have the following line

<listen_host>0.0.0.0</listen_host>

in the clickhouse/config.xml.

5

Sharing your full install logs would help if you suspect snuba bootstrap might have failed in the middle. That said the error message seems to suggest this is deeper.

I think we can force Clickhouse to run in IPv4 mode by adding that line in the config file.

Can you share your clickhouse/config.xml file?

6 
<yandex>
    <listen_host>0.0.0.0</listen_host>
    <max_server_memory_usage_to_ram_ratio from_env="MAX_MEMORY_USAGE_RATIO" />
    <logger>
        <level>information</level>
        <console>1</console>
    </logger>
    <merge_tree>
        <enable_mixed_granularity_parts>1</enable_mixed_granularity_parts>
    </merge_tree>
</yandex>
7

Unfortunately, I don’t have the full logs now. I have been doing a lot of reinstall lately and trying out different configurations.

8

Well, if you can get us one that demonstrates the issue we can help. Otherwise it is hard to move forward from this point.

9

I wish I could. The logs shows up from clickhouse container, when running docker-compose up -d. I missed the oppertunity to save the logs then. Please close the topic. If I see again, I will report it.

1 Like
10

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.