运行 runner 脚本无法在 Clickhouse 中进行集成测试
Failed to do integration test in Clickhouse by running with runner script
我试图通过 docker 从此 README.md 进行 运行 集成测试，但是 运行 遇到了一些问题。
这是我的开发环境：
[boblee@Wsl[19:50:36]server]$ sudo docker version
Client:
Version: 17.06.1-ce-rc1
EulerVersion: 17.06.0.10
API version: 1.30
Go version: go1.8.3
Git commit: 7a547fb
Built: Fri Mar 23 05:39:01 2018
OS/Arch: linux/amd64
Server:
Version: 17.06.1-ce-rc1
EulerVersion: 17.06.0.10
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: 7a547fb
Built: Fri Mar 23 05:38:32 2018
OS/Arch: linux/amd64
Experimental: false
我使用 docker pull yandex/clickhouse-integration-tests-runner 并从 Yandex 获取推荐图片。
[boblee@Wsl[19:52:10]server]$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
yandex/clickhouse-integration-tests-runner latest 811ba8d0c7f3 5 days ago 1.49GB
The only requirement is and docker pull yandex/clickhouse-integration-tests-runner
所以，我的问题是：fresh configured docker 是什么意思？
我可以假设拉取的 docker 图像已经具有用于集成测试的所有第三方依赖项吗？
然后：
我按照命令去了 $CLICKHOUSE_HOME/tests/integration 和 运行，也是从 README.md
sudo ./runner --binary /home/boblee/ClickHouse/build_20.13.1.1_debug/programs/clickhouse --bridge-binary /home/boblee/ClickHouse/build_20.13.1.1_debug/programs/clickhouse-odbc-bridge --base-configs-dir /home/boblee/ClickHouse/build_20.13.1.1_debug/programs/server/ 'test_odbc_interaction -ss'
部分错误：
Start tests
======================================================================================================== test session starts =========================================================================================================platform linux -- Python 3.6.9, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /ClickHouse/tests/integration, configfile: pytest.ini
plugins: timeout-1.4.2
timeout: 300.0s
timeout method: signal
timeout func_only: False
collecting ... ENV DOCKER_KERBEROS_KDC_TAG latest
ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse
ENV HOSTNAME DESKTOP-MB3AOEA
ENV SHLVL 0
ENV PYTEST_OPTS test_odbc_interaction -ss
ENV OLDPWD /
ENV HOME /root
ENV DOCKER_MYSQL_PHP_CLIENT_TAG latest
ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse
ENV DOCKER_MYSQL_JS_CLIENT_TAG latest
ENV TERM xterm
ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV DOCKER_CHANNEL stable
ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG latest
ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG latest
ENV DOCKER_VERSION 5:19.03.13~3-0~ubuntu-bionic
ENV PWD /ClickHouse/tests/integration
ENV DOCKER_MYSQL_JAVA_CLIENT_TAG latest
ENV CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge
ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config
ENV TZ Europe/Moscow
CLUSTER INIT base_config_dir:/clickhouse-config
collected 10 items
test_odbc_interaction/test.py Cluster start called. is_up=False, destroy_dirs=True
Trying to kill unstopped containers...
ERROR:
Can't find a suitable configuration file in this directory or any
parent. Are you in the right directory?
Supported filenames: docker-compose.yml, docker-compose.yaml
ERROR:
Can't find a suitable configuration file in this directory or any
parent. Are you in the right directory?
Supported filenames: docker-compose.yml, docker-compose.yaml
Unstopped containers killed
('Removing instances dir %s', '/ClickHouse/tests/integration/test_odbc_interaction/_instances')
Setup directory for instance: node1 destroy_dirs: True
Copy common default production configuration from /clickhouse-config
Create directory for configuration generated in this helper
Create directory for common tests configuration
Copy common configuration from helpers
Generate and write macros file
Copy custom test config files ['/ClickHouse/tests/integration/test_odbc_interaction/configs/openssl.xml', '/ClickHouse/tests/integration/test_odbc_interaction/configs/odbc_logging.xml', '/ClickHouse/tests/integration/test_odbc_interaction/configs/enable_dictionaries.xml', '/ClickHouse/tests/integration/test_odbc_interaction/configs/dictionaries/sqlite3_odbc_hashed_dictionary.xml', '/ClickHouse/tests/integration/test_odbc_interaction/configs/dictionaries/sqlite3_odbc_cached_dictionary.xml', '/ClickHouse/tests/integration/test_odbc_interaction/configs/dictionaries/postgres_odbc_hashed_dictionary.xml'] to /ClickHouse/tests/integration/test_odbc_interaction/_instances/node1/configs/config.d
Setup database dir /ClickHouse/tests/integration/test_odbc_interaction/_instances/node1/database
Setup logs dir /ClickHouse/tests/integration/test_odbc_interaction/_instances/node1/logs
Env {'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_odbc_interaction/_instances/node1/env_file
Entrypoint cmd: bash -c "clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; tail -f /dev/null"
Setup MySQL
Stderr:
b'Creating network "roottestodbcinteraction_default" with the default driver\nPulling mysql1 (mysql:5.7)...\nGet https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n'
Stdout:
b''
Failed to start cluster:
Command ['docker-compose', '--project-directory', '/ClickHouse/tests/integration/test_odbc_interaction', '--project-name', 'roottestodbcinteraction', '--file', '/compose/docker_compose_mysql.yml', 'up', '-d', '--force-recreate'] return non-zero code 1: b'Creating network "roottestodbcinteraction_default" with the default driver\nPulling mysql1 (mysql:5.7)...\nGet https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n'
Traceback (most recent call last):
File "/ClickHouse/tests/integration/helpers/cluster.py", line 655, in start
subprocess_check_call(self.base_mysql_cmd + common_opts)
File "/ClickHouse/tests/integration/helpers/cluster.py", line 60, in subprocess_check_call
run_and_check(args)
File "/ClickHouse/tests/integration/helpers/cluster.py", line 54, in run_and_check
raise Exception('Command {} return non-zero code {}: {}'.format(args, res.returncode, res.stderr))
Exception: Command ['docker-compose', '--project-directory', '/ClickHouse/tests/integration/test_odbc_interaction', '--project-name', 'roottestodbcinteraction', '--file', '/compose/docker_compose_mysql.yml', 'up', '-d', '--force-recreate'] return non-zero code 1: b'Creating network "roottestodbcinteraction_default" with the default driver\nPulling mysql1 (mysql:5.7)...\nGet https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n'
None
Command ['docker-compose', '--project-directory', '/ClickHouse/tests/integration/test_odbc_interaction', '--project-name', 'roottestodbcinteraction', '--file', '/compose/docker_compose_mysql.yml', 'up', '-d', '--force-recreate'] return non-zero code 1: b'Creating network "roottestodbcinteraction_default" with the default driver\nPulling mysql1 (mysql:5.7)...\nGet https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n'
EEEEEEEEEE
=============================================================================================================== ERRORS ===============================================================================================================__________________________________________________________________________________________ ERROR at setup of test_mysql_simple_select_works __________________________________________________________________________________________
@pytest.fixture(scope="module")
def started_cluster():
try:
cluster.start()
sqlite_db = node1.odbc_drivers["SQLite3"]["Database"]
print("sqlite data received")
node1.exec_in_container(
["bash", "-c", "echo 'CREATE TABLE t1(x INTEGER PRIMARY KEY ASC, y, z);' | sqlite3 {}".format(sqlite_db)],
privileged=True, user='root')
node1.exec_in_container(
["bash", "-c", "echo 'CREATE TABLE t2(X INTEGER PRIMARY KEY ASC, Y, Z);' | sqlite3 {}".format(sqlite_db)],
privileged=True, user='root')
node1.exec_in_container(
["bash", "-c", "echo 'CREATE TABLE t3(X INTEGER PRIMARY KEY ASC, Y, Z);' | sqlite3 {}".format(sqlite_db)],
privileged=True, user='root')
node1.exec_in_container(
["bash", "-c", "echo 'CREATE TABLE t4(X INTEGER PRIMARY KEY ASC, Y, Z);' | sqlite3 {}".format(sqlite_db)],
privileged=True, user='root')
print("sqlite tables created")
mysql_conn = get_mysql_conn()
print("mysql connection received")
## create mysql db and table
create_mysql_db(mysql_conn, 'clickhouse')
print("mysql database created")
postgres_conn = get_postgres_conn()
print("postgres connection received")
create_postgres_db(postgres_conn, 'clickhouse')
print("postgres db created")
cursor = postgres_conn.cursor()
cursor.execute(
"create table if not exists clickhouse.test_table (column1 int primary key, column2 varchar(40) not null)")
yield cluster
except Exception as ex:
print(ex)
> raise ex
test_odbc_interaction/test.py:98:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
目前， docker 似乎无法正确 运行 因为找不到 docker-compose.yaml
ERROR:
Can't find a suitable configuration file in this directory or any
parent. Are you in the right directory?
Supported filenames: docker-compose.yml, docker-compose.yaml
有什么想法吗？
我解决了问题，回来分享我的解决方案。
这个问题可能是网络环境造成的。如果您为 Docker 配置了代理，更改代理或简单地手动拉取图像可能会有所帮助（您可以从 $ClickHouse/docker/test/integration/runner/compose/xxx.yml).
中找到所需的图像
我试图通过 docker 从此 README.md 进行 运行 集成测试，但是 运行 遇到了一些问题。
这是我的开发环境：
[boblee@Wsl[19:50:36]server]$ sudo docker version
Client:
Version: 17.06.1-ce-rc1
EulerVersion: 17.06.0.10
API version: 1.30
Go version: go1.8.3
Git commit: 7a547fb
Built: Fri Mar 23 05:39:01 2018
OS/Arch: linux/amd64
Server:
Version: 17.06.1-ce-rc1
EulerVersion: 17.06.0.10
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: 7a547fb
Built: Fri Mar 23 05:38:32 2018
OS/Arch: linux/amd64
Experimental: false
我使用 docker pull yandex/clickhouse-integration-tests-runner 并从 Yandex 获取推荐图片。
[boblee@Wsl[19:52:10]server]$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
yandex/clickhouse-integration-tests-runner latest 811ba8d0c7f3 5 days ago 1.49GB
The only requirement is and docker pull yandex/clickhouse-integration-tests-runner
所以，我的问题是：fresh configured docker 是什么意思？
我可以假设拉取的 docker 图像已经具有用于集成测试的所有第三方依赖项吗？
然后：
我按照命令去了 $CLICKHOUSE_HOME/tests/integration 和 运行，也是从 README.md
sudo ./runner --binary /home/boblee/ClickHouse/build_20.13.1.1_debug/programs/clickhouse --bridge-binary /home/boblee/ClickHouse/build_20.13.1.1_debug/programs/clickhouse-odbc-bridge --base-configs-dir /home/boblee/ClickHouse/build_20.13.1.1_debug/programs/server/ 'test_odbc_interaction -ss'
部分错误：
Start tests
======================================================================================================== test session starts =========================================================================================================platform linux -- Python 3.6.9, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /ClickHouse/tests/integration, configfile: pytest.ini
plugins: timeout-1.4.2
timeout: 300.0s
timeout method: signal
timeout func_only: False
collecting ... ENV DOCKER_KERBEROS_KDC_TAG latest
ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse
ENV HOSTNAME DESKTOP-MB3AOEA
ENV SHLVL 0
ENV PYTEST_OPTS test_odbc_interaction -ss
ENV OLDPWD /
ENV HOME /root
ENV DOCKER_MYSQL_PHP_CLIENT_TAG latest
ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse
ENV DOCKER_MYSQL_JS_CLIENT_TAG latest
ENV TERM xterm
ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV DOCKER_CHANNEL stable
ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG latest
ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG latest
ENV DOCKER_VERSION 5:19.03.13~3-0~ubuntu-bionic
ENV PWD /ClickHouse/tests/integration
ENV DOCKER_MYSQL_JAVA_CLIENT_TAG latest
ENV CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge
ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config
ENV TZ Europe/Moscow
CLUSTER INIT base_config_dir:/clickhouse-config
collected 10 items
test_odbc_interaction/test.py Cluster start called. is_up=False, destroy_dirs=True
Trying to kill unstopped containers...
ERROR:
Can't find a suitable configuration file in this directory or any
parent. Are you in the right directory?
Supported filenames: docker-compose.yml, docker-compose.yaml
ERROR:
Can't find a suitable configuration file in this directory or any
parent. Are you in the right directory?
Supported filenames: docker-compose.yml, docker-compose.yaml
Unstopped containers killed
('Removing instances dir %s', '/ClickHouse/tests/integration/test_odbc_interaction/_instances')
Setup directory for instance: node1 destroy_dirs: True
Copy common default production configuration from /clickhouse-config
Create directory for configuration generated in this helper
Create directory for common tests configuration
Copy common configuration from helpers
Generate and write macros file
Copy custom test config files ['/ClickHouse/tests/integration/test_odbc_interaction/configs/openssl.xml', '/ClickHouse/tests/integration/test_odbc_interaction/configs/odbc_logging.xml', '/ClickHouse/tests/integration/test_odbc_interaction/configs/enable_dictionaries.xml', '/ClickHouse/tests/integration/test_odbc_interaction/configs/dictionaries/sqlite3_odbc_hashed_dictionary.xml', '/ClickHouse/tests/integration/test_odbc_interaction/configs/dictionaries/sqlite3_odbc_cached_dictionary.xml', '/ClickHouse/tests/integration/test_odbc_interaction/configs/dictionaries/postgres_odbc_hashed_dictionary.xml'] to /ClickHouse/tests/integration/test_odbc_interaction/_instances/node1/configs/config.d
Setup database dir /ClickHouse/tests/integration/test_odbc_interaction/_instances/node1/database
Setup logs dir /ClickHouse/tests/integration/test_odbc_interaction/_instances/node1/logs
Env {'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_odbc_interaction/_instances/node1/env_file
Entrypoint cmd: bash -c "clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; tail -f /dev/null"
Setup MySQL
Stderr:
b'Creating network "roottestodbcinteraction_default" with the default driver\nPulling mysql1 (mysql:5.7)...\nGet https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n'
Stdout:
b''
Failed to start cluster:
Command ['docker-compose', '--project-directory', '/ClickHouse/tests/integration/test_odbc_interaction', '--project-name', 'roottestodbcinteraction', '--file', '/compose/docker_compose_mysql.yml', 'up', '-d', '--force-recreate'] return non-zero code 1: b'Creating network "roottestodbcinteraction_default" with the default driver\nPulling mysql1 (mysql:5.7)...\nGet https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n'
Traceback (most recent call last):
File "/ClickHouse/tests/integration/helpers/cluster.py", line 655, in start
subprocess_check_call(self.base_mysql_cmd + common_opts)
File "/ClickHouse/tests/integration/helpers/cluster.py", line 60, in subprocess_check_call
run_and_check(args)
File "/ClickHouse/tests/integration/helpers/cluster.py", line 54, in run_and_check
raise Exception('Command {} return non-zero code {}: {}'.format(args, res.returncode, res.stderr))
Exception: Command ['docker-compose', '--project-directory', '/ClickHouse/tests/integration/test_odbc_interaction', '--project-name', 'roottestodbcinteraction', '--file', '/compose/docker_compose_mysql.yml', 'up', '-d', '--force-recreate'] return non-zero code 1: b'Creating network "roottestodbcinteraction_default" with the default driver\nPulling mysql1 (mysql:5.7)...\nGet https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n'
None
Command ['docker-compose', '--project-directory', '/ClickHouse/tests/integration/test_odbc_interaction', '--project-name', 'roottestodbcinteraction', '--file', '/compose/docker_compose_mysql.yml', 'up', '-d', '--force-recreate'] return non-zero code 1: b'Creating network "roottestodbcinteraction_default" with the default driver\nPulling mysql1 (mysql:5.7)...\nGet https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n'
EEEEEEEEEE
=============================================================================================================== ERRORS ===============================================================================================================__________________________________________________________________________________________ ERROR at setup of test_mysql_simple_select_works __________________________________________________________________________________________
@pytest.fixture(scope="module")
def started_cluster():
try:
cluster.start()
sqlite_db = node1.odbc_drivers["SQLite3"]["Database"]
print("sqlite data received")
node1.exec_in_container(
["bash", "-c", "echo 'CREATE TABLE t1(x INTEGER PRIMARY KEY ASC, y, z);' | sqlite3 {}".format(sqlite_db)],
privileged=True, user='root')
node1.exec_in_container(
["bash", "-c", "echo 'CREATE TABLE t2(X INTEGER PRIMARY KEY ASC, Y, Z);' | sqlite3 {}".format(sqlite_db)],
privileged=True, user='root')
node1.exec_in_container(
["bash", "-c", "echo 'CREATE TABLE t3(X INTEGER PRIMARY KEY ASC, Y, Z);' | sqlite3 {}".format(sqlite_db)],
privileged=True, user='root')
node1.exec_in_container(
["bash", "-c", "echo 'CREATE TABLE t4(X INTEGER PRIMARY KEY ASC, Y, Z);' | sqlite3 {}".format(sqlite_db)],
privileged=True, user='root')
print("sqlite tables created")
mysql_conn = get_mysql_conn()
print("mysql connection received")
## create mysql db and table
create_mysql_db(mysql_conn, 'clickhouse')
print("mysql database created")
postgres_conn = get_postgres_conn()
print("postgres connection received")
create_postgres_db(postgres_conn, 'clickhouse')
print("postgres db created")
cursor = postgres_conn.cursor()
cursor.execute(
"create table if not exists clickhouse.test_table (column1 int primary key, column2 varchar(40) not null)")
yield cluster
except Exception as ex:
print(ex)
> raise ex
test_odbc_interaction/test.py:98:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
目前， docker 似乎无法正确 运行 因为找不到 docker-compose.yaml
ERROR: Can't find a suitable configuration file in this directory or any parent. Are you in the right directory? Supported filenames: docker-compose.yml, docker-compose.yaml
有什么想法吗？
我解决了问题，回来分享我的解决方案。
这个问题可能是网络环境造成的。如果您为 Docker 配置了代理，更改代理或简单地手动拉取图像可能会有所帮助（您可以从 $ClickHouse/docker/test/integration/runner/compose/xxx.yml).