You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/10/07 15:37:45 UTC
[GitHub] [doris] morningman opened a new pull request, #13143: [test](jdbc) add jdbc regression test
morningman opened a new pull request, #13143:
URL: https://github.com/apache/doris/pull/13143
# Proposed changes
Issue Number: close #xxx
## Problem summary
1. Modify default behavior of `build.sh`
The `BUILD_JAVA_UDF` is default ON, so that jvm is needed for compilation and runtime.
2. Add docker-compose for MySQL 5.7 and PostgreSQL 14
See `docker/thirdparties/docker-compose`.
3. Add some regression test cases for jdbc query on MySQL and PG
The default is `false`, if set to true, you need first start docker for MySQL and PG.
4. Support `if not exists` and `if exists` for create/drop resource and create/drop encryptkey
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
6. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
7. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
8. Does it need to update dependencies:
- [ ] Yes
- [ ] No
9. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13143: [test](jdbc) add jdbc regression test
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13143:
URL: https://github.com/apache/doris/pull/13143#issuecomment-1271753206
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/3205943559") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In docker/thirdparties/docker-compose/mysql/mysql-5.7-env.sh line 1:
DOCKER_MYSQL_57_INTERNAL_PORT=3306
^-- SC2148 (error): Tips depend on target shell and yours is unknown. Add a shebang or a 'shell' directive.
^---------------------------^ SC2034 (warning): DOCKER_MYSQL_57_INTERNAL_PORT appears unused. Verify use (or export if used externally).
In docker/thirdparties/docker-compose/mysql/mysql-5.7-env.sh line 2:
DOCKER_MYSQL_57_EXTERNAL_PORT=3316
^---------------------------^ SC2034 (warning): DOCKER_MYSQL_57_EXTERNAL_PORT appears unused. Verify use (or export if used externally).
In docker/thirdparties/docker-compose/mysql/mysql-5.7-env.sh line 3:
DOCKER_MYSQL_57_DATA_DIR=./data/
^----------------------^ SC2034 (warning): DOCKER_MYSQL_57_DATA_DIR appears unused. Verify use (or export if used externally).
In docker/thirdparties/docker-compose/postgresql/postgresql-14-env.sh line 1:
DOCKER_PG_14_INTERNAL_PORT=5432
^-- SC2148 (error): Tips depend on target shell and yours is unknown. Add a shebang or a 'shell' directive.
^------------------------^ SC2034 (warning): DOCKER_PG_14_INTERNAL_PORT appears unused. Verify use (or export if used externally).
In docker/thirdparties/docker-compose/postgresql/postgresql-14-env.sh line 2:
DOCKER_PG_14_EXTERNAL_PORT=5442
^------------------------^ SC2034 (warning): DOCKER_PG_14_EXTERNAL_PORT appears unused. Verify use (or export if used externally).
In docker/thirdparties/docker-compose/postgresql/postgresql-14-env.sh line 3:
DOCKER_PG_14_DATA_DIR=./data/data/
^-------------------^ SC2034 (warning): DOCKER_PG_14_DATA_DIR appears unused. Verify use (or export if used externally).
In docker/thirdparties/start-thirdparties-docker.sh line 28:
docker compose -f ${ROOT}/docker-compose/elasticsearch/elasticsearch.yaml down
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
docker compose -f "${ROOT}"/docker-compose/elasticsearch/elasticsearch.yaml down
In docker/thirdparties/start-thirdparties-docker.sh line 29:
docker compose -f ${ROOT}/docker-compose/elasticsearch/elasticsearch.yaml up -d
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
docker compose -f "${ROOT}"/docker-compose/elasticsearch/elasticsearch.yaml up -d
In docker/thirdparties/start-thirdparties-docker.sh line 32:
mkdir -p ${ROOT}/docker-compose/mysql/data/
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mkdir -p "${ROOT}"/docker-compose/mysql/data/
In docker/thirdparties/start-thirdparties-docker.sh line 33:
rm ${ROOT}/docker-compose/mysql/data/* -rf
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
rm "${ROOT}"/docker-compose/mysql/data/* -rf
In docker/thirdparties/start-thirdparties-docker.sh line 34:
docker compose -f ${ROOT}/docker-compose/mysql/mysql-5.7.yaml --env-file ${ROOT}/docker-compose/mysql/mysql-5.7-env.sh down
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
docker compose -f "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml --env-file "${ROOT}"/docker-compose/mysql/mysql-5.7-env.sh down
In docker/thirdparties/start-thirdparties-docker.sh line 35:
docker compose -f ${ROOT}/docker-compose/mysql/mysql-5.7.yaml --env-file ${ROOT}/docker-compose/mysql/mysql-5.7-env.sh up -d
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
docker compose -f "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml --env-file "${ROOT}"/docker-compose/mysql/mysql-5.7-env.sh up -d
In docker/thirdparties/start-thirdparties-docker.sh line 38:
mkdir -p ${ROOT}/docker-compose/postgresql/data/data
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mkdir -p "${ROOT}"/docker-compose/postgresql/data/data
In docker/thirdparties/start-thirdparties-docker.sh line 39:
rm ${ROOT}/docker-compose/postgresql/data/data/* -rf
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
rm "${ROOT}"/docker-compose/postgresql/data/data/* -rf
In docker/thirdparties/start-thirdparties-docker.sh line 40:
docker compose -f ${ROOT}/docker-compose/postgresql/postgresql-14.yaml --env-file ${ROOT}/docker-compose/postgresql/postgresql-14-env.sh down
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
docker compose -f "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml --env-file "${ROOT}"/docker-compose/postgresql/postgresql-14-env.sh down
In docker/thirdparties/start-thirdparties-docker.sh line 41:
docker compose -f ${ROOT}/docker-compose/postgresql/postgresql-14.yaml --env-file ${ROOT}/docker-compose/postgresql/postgresql-14-env.sh up -d
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
docker compose -f "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml --env-file "${ROOT}"/docker-compose/postgresql/postgresql-14-env.sh up -d
For more information:
https://www.shellcheck.net/wiki/SC2148 -- Tips depend on target shell and y...
https://www.shellcheck.net/wiki/SC2034 -- DOCKER_MYSQL_57_DATA_DIR appears ...
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' found no issues.
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13143: [test](jdbc) add jdbc and hive regression test
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13143:
URL: https://github.com/apache/doris/pull/13143#issuecomment-1272565640
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/3214629210") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In docker/thirdparties/docker-compose/hive/gen_env.sh line 35:
echo "FS_PORT=${FS_PORT}" >>"${ROOT}"/hadoop-hive.env
^-- SC2129 (style): Consider using { cmd1; cmd2; } >> file instead of individual redirects.
For more information:
https://www.shellcheck.net/wiki/SC2129 -- Consider using { cmd1; cmd2; } >>...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' found no issues.
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman merged pull request #13143: [test](jdbc) add jdbc and hive regression test
Posted by GitBox <gi...@apache.org>.
morningman merged PR #13143:
URL: https://github.com/apache/doris/pull/13143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Ray-Eldath commented on a diff in pull request #13143: [test](jdbc) add jdbc and hive regression test
Posted by GitBox <gi...@apache.org>.
Ray-Eldath commented on code in PR #13143:
URL: https://github.com/apache/doris/pull/13143#discussion_r994594093
##########
docker/thirdparties/start-thirdparties-docker.sh:
##########
@@ -20,8 +20,38 @@
# This script will restart all thirdparty containers
################################################################
-################################################################
-# restart elasticsearch containers
-################################################################
-docker compose -f docker-compose/elasticsearch.yaml down
-docker compose -f docker-compose/elasticsearch.yaml up -d
+set -eo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" &>/dev/null && pwd)"
+
+# If you want to start multi group of these containers on same host,
+# Change this to a specific string.
+# Do not use "_" or other sepcial characters, only number and alphabeta.
+# NOTICE: change this uid will modify the file in docker-compose.
+CONTAINER_UID="doris--"
+
+# elasticsearch
+# docker compose -f "${ROOT}"/docker-compose/elasticsearch/elasticsearch.yaml down
+# docker compose -f "${ROOT}"/docker-compose/elasticsearch/elasticsearch.yaml up -d
+
+# mysql 5.7
+sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml
+sudo docker compose -f "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml --env-file "${ROOT}"/docker-compose/mysql/mysql-5.7.env down
+sudo mkdir -p "${ROOT}"/docker-compose/mysql/data/
+sudo rm "${ROOT}"/docker-compose/mysql/data/* -rf
+sudo docker compose -f "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml --env-file "${ROOT}"/docker-compose/mysql/mysql-5.7.env up -d
+
+# pg 14
+sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml
+sudo docker compose -f "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml --env-file "${ROOT}"/docker-compose/postgresql/postgresql-14.env down
+sudo mkdir -p "${ROOT}"/docker-compose/postgresql/data/data
+sudo rm "${ROOT}"/docker-compose/postgresql/data/data/* -rf
+sudo docker compose -f "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml --env-file "${ROOT}"/docker-compose/postgresql/postgresql-14.env up -d
+
+# hive
+# before start it, you need to download parquet file package, see "README" in "docker-compose/hive/scripts/"
+sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/hive/hive-2x.yaml
+sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/hive/hadoop-hive.env.tpl
+sudo docker compose -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive.env down
+sudo sh "${ROOT}"/docker-compose/hive/gen_env.sh
+sudo docker compose -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive.env up -d
Review Comment:
these should be `docker-compose` not `docker compose` I suppose?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Ray-Eldath commented on a diff in pull request #13143: [test](jdbc) add jdbc and hive regression test
Posted by GitBox <gi...@apache.org>.
Ray-Eldath commented on code in PR #13143:
URL: https://github.com/apache/doris/pull/13143#discussion_r994609133
##########
regression-test/suites/jdbc_p0/test_jdbc_query_mysql.groovy:
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_jdbc_query_mysql", "p0") {
+
+ String enabled = context.config.otherConfigs.get("enableJdbcTest")
+ if (enabled != null && enabled.equalsIgnoreCase("true")) {
+ String mysql_57_port = context.config.otherConfigs.get("mysql_57_port")
+ String jdbcResourceMysql57 = "jdbc_resource_mysql_57"
+ String jdbcMysql57Table1 = "jdbc_mysql_57_table1"
+
+ sql """drop resource if exists $jdbcResourceMysql57;"""
+ sql """
+ create external resource $jdbcResourceMysql57
+ properties (
+ "type"="jdbc",
+ "user"="root",
+ "password"="123456",
+ "jdbc_url"="jdbc:mysql://127.0.0.1:3316/doris_test",
Review Comment:
```suggestion
"jdbc_url"="jdbc:mysql://127.0.0.1:$mysql_57_port/doris_test",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13143: [test](jdbc) add jdbc regression test
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13143:
URL: https://github.com/apache/doris/pull/13143#issuecomment-1272460348
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/3213008669") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In docker/thirdparties/docker-compose/hive/gen_env.sh line 27:
cp ${ROOT}/hadoop-hive.env.tpl ${ROOT}/hadoop-hive.env
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
cp "${ROOT}"/hadoop-hive.env.tpl "${ROOT}"/hadoop-hive.env
In docker/thirdparties/docker-compose/hive/gen_env.sh line 30:
HOST_NAME=`hostname`
^--------^ SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
Did you mean:
HOST_NAME=$(hostname)
In docker/thirdparties/docker-compose/hive/gen_env.sh line 31:
echo "CORE_CONF_fs_defaultFS=hdfs://${HOST_NAME}:8020" >> ${ROOT}/hadoop-hive.env
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
echo "CORE_CONF_fs_defaultFS=hdfs://${HOST_NAME}:8020" >> "${ROOT}"/hadoop-hive.env
In docker/thirdparties/docker-compose/hive/gen_env.sh line 32:
echo "HOST_NAME=${HOST_NAME}" >> ${ROOT}/hadoop-hive.env
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
echo "HOST_NAME=${HOST_NAME}" >> "${ROOT}"/hadoop-hive.env
In docker/thirdparties/docker-compose/hive/scripts/healthy_check.sh line 19:
if [ ! -f "/mnt/SUCCESS" ]; then
^---------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
Did you mean:
if [[ ! -f "/mnt/SUCCESS" ]]; then
In docker/thirdparties/docker-compose/hive/scripts/hive-metastore.sh line 21:
if [ ! -d "/mnt/scritps/tpch1.db" ];then
^-- SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
Did you mean:
if [[ ! -d "/mnt/scritps/tpch1.db" ]];then
For more information:
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
https://www.shellcheck.net/wiki/SC2006 -- Use $(...) notation instead of le...
https://www.shellcheck.net/wiki/SC2292 -- Prefer [[ ]] over [ ] for tests i...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' returned error 1 finding the following formatting issues:
----------
--- docker/thirdparties/docker-compose/hive/gen_env.sh.orig
+++ docker/thirdparties/docker-compose/hive/gen_env.sh
@@ -27,6 +27,6 @@
cp ${ROOT}/hadoop-hive.env.tpl ${ROOT}/hadoop-hive.env
# Need to set hostname of container to same as host machine's.
# Otherwise, the doris process can not connect to namenode directly.
-HOST_NAME=`hostname`
-echo "CORE_CONF_fs_defaultFS=hdfs://${HOST_NAME}:8020" >> ${ROOT}/hadoop-hive.env
-echo "HOST_NAME=${HOST_NAME}" >> ${ROOT}/hadoop-hive.env
+HOST_NAME=$(hostname)
+echo "CORE_CONF_fs_defaultFS=hdfs://${HOST_NAME}:8020" >>${ROOT}/hadoop-hive.env
+echo "HOST_NAME=${HOST_NAME}" >>${ROOT}/hadoop-hive.env
--- docker/thirdparties/docker-compose/hive/scripts/healthy_check.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/healthy_check.sh
@@ -21,4 +21,3 @@
else
exit 0
fi
-
--- docker/thirdparties/docker-compose/hive/scripts/hive-metastore.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/hive-metastore.sh
@@ -18,7 +18,7 @@
nohup /opt/hive/bin/hive --service metastore &
sleep 10
-if [ ! -d "/mnt/scritps/tpch1.db" ];then
+if [ ! -d "/mnt/scritps/tpch1.db" ]; then
echo "/mnt/scritps/tpch1.db does not exist"
exit 1
fi
@@ -30,6 +30,6 @@
touch /mnt/SUCCESS
# Avoid container exit
-while true;do
+while true; do
sleep 1
done
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Ray-Eldath commented on a diff in pull request #13143: [test](jdbc) add jdbc and hive regression test
Posted by GitBox <gi...@apache.org>.
Ray-Eldath commented on code in PR #13143:
URL: https://github.com/apache/doris/pull/13143#discussion_r994609441
##########
regression-test/suites/jdbc_p0/test_jdbc_query_pg.groovy:
##########
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_jdbc_query_pg", "p0") {
+
+ String enabled = context.config.otherConfigs.get("enableJdbcTest")
+ if (enabled != null && enabled.equalsIgnoreCase("true")) {
+ String pg_14_port = context.config.otherConfigs.get("pg_14_port")
+ String jdbcResourcePg14 = "jdbc_resource_pg_14"
+ String jdbcPg14Table1 = "jdbc_pg_14_table1"
+
+ sql """drop resource if exists $jdbcResourcePg14;"""
+ sql """
+ create external resource $jdbcResourcePg14
+ properties (
+ "type"="jdbc",
+ "user"="postgres",
+ "password"="123456",
+ "jdbc_url"="jdbc:postgresql://127.0.0.1:5442/postgres?currentSchema=doris_test",
Review Comment:
```suggestion
"jdbc_url"="jdbc:postgresql://127.0.0.1:$pg_14_port/postgres?currentSchema=doris_test",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Ray-Eldath commented on a diff in pull request #13143: [test](jdbc) add jdbc and hive regression test
Posted by GitBox <gi...@apache.org>.
Ray-Eldath commented on code in PR #13143:
URL: https://github.com/apache/doris/pull/13143#discussion_r994594093
##########
docker/thirdparties/start-thirdparties-docker.sh:
##########
@@ -20,8 +20,38 @@
# This script will restart all thirdparty containers
################################################################
-################################################################
-# restart elasticsearch containers
-################################################################
-docker compose -f docker-compose/elasticsearch.yaml down
-docker compose -f docker-compose/elasticsearch.yaml up -d
+set -eo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" &>/dev/null && pwd)"
+
+# If you want to start multi group of these containers on same host,
+# Change this to a specific string.
+# Do not use "_" or other sepcial characters, only number and alphabeta.
+# NOTICE: change this uid will modify the file in docker-compose.
+CONTAINER_UID="doris--"
+
+# elasticsearch
+# docker compose -f "${ROOT}"/docker-compose/elasticsearch/elasticsearch.yaml down
+# docker compose -f "${ROOT}"/docker-compose/elasticsearch/elasticsearch.yaml up -d
+
+# mysql 5.7
+sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml
+sudo docker compose -f "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml --env-file "${ROOT}"/docker-compose/mysql/mysql-5.7.env down
+sudo mkdir -p "${ROOT}"/docker-compose/mysql/data/
+sudo rm "${ROOT}"/docker-compose/mysql/data/* -rf
+sudo docker compose -f "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml --env-file "${ROOT}"/docker-compose/mysql/mysql-5.7.env up -d
+
+# pg 14
+sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml
+sudo docker compose -f "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml --env-file "${ROOT}"/docker-compose/postgresql/postgresql-14.env down
+sudo mkdir -p "${ROOT}"/docker-compose/postgresql/data/data
+sudo rm "${ROOT}"/docker-compose/postgresql/data/data/* -rf
+sudo docker compose -f "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml --env-file "${ROOT}"/docker-compose/postgresql/postgresql-14.env up -d
+
+# hive
+# before start it, you need to download parquet file package, see "README" in "docker-compose/hive/scripts/"
+sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/hive/hive-2x.yaml
+sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/hive/hadoop-hive.env.tpl
+sudo docker compose -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive.env down
+sudo sh "${ROOT}"/docker-compose/hive/gen_env.sh
+sudo docker compose -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive.env up -d
Review Comment:
these should be `docker-compose` not `docker compose` I suppose?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Ray-Eldath commented on a diff in pull request #13143: [test](jdbc) add jdbc and hive regression test
Posted by GitBox <gi...@apache.org>.
Ray-Eldath commented on code in PR #13143:
URL: https://github.com/apache/doris/pull/13143#discussion_r994796842
##########
docker/thirdparties/start-thirdparties-docker.sh:
##########
@@ -20,8 +20,38 @@
# This script will restart all thirdparty containers
################################################################
-################################################################
-# restart elasticsearch containers
-################################################################
-docker compose -f docker-compose/elasticsearch.yaml down
-docker compose -f docker-compose/elasticsearch.yaml up -d
+set -eo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" &>/dev/null && pwd)"
+
+# If you want to start multi group of these containers on same host,
+# Change this to a specific string.
+# Do not use "_" or other sepcial characters, only number and alphabeta.
+# NOTICE: change this uid will modify the file in docker-compose.
+CONTAINER_UID="doris--"
+
+# elasticsearch
+# docker compose -f "${ROOT}"/docker-compose/elasticsearch/elasticsearch.yaml down
+# docker compose -f "${ROOT}"/docker-compose/elasticsearch/elasticsearch.yaml up -d
+
+# mysql 5.7
+sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml
+sudo docker compose -f "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml --env-file "${ROOT}"/docker-compose/mysql/mysql-5.7.env down
+sudo mkdir -p "${ROOT}"/docker-compose/mysql/data/
+sudo rm "${ROOT}"/docker-compose/mysql/data/* -rf
+sudo docker compose -f "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml --env-file "${ROOT}"/docker-compose/mysql/mysql-5.7.env up -d
+
+# pg 14
+sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml
+sudo docker compose -f "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml --env-file "${ROOT}"/docker-compose/postgresql/postgresql-14.env down
+sudo mkdir -p "${ROOT}"/docker-compose/postgresql/data/data
+sudo rm "${ROOT}"/docker-compose/postgresql/data/data/* -rf
+sudo docker compose -f "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml --env-file "${ROOT}"/docker-compose/postgresql/postgresql-14.env up -d
+
+# hive
+# before start it, you need to download parquet file package, see "README" in "docker-compose/hive/scripts/"
+sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/hive/hive-2x.yaml
+sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/hive/hadoop-hive.env.tpl
+sudo docker compose -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive.env down
+sudo sh "${ROOT}"/docker-compose/hive/gen_env.sh
Review Comment:
```suggestion
sudo bash "${ROOT}"/docker-compose/hive/gen_env.sh
sudo docker compose -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive.env down
```
reverse these two lines, then replace `sh` to `bash`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Ray-Eldath commented on a diff in pull request #13143: [test](jdbc) add jdbc and hive regression test
Posted by GitBox <gi...@apache.org>.
Ray-Eldath commented on code in PR #13143:
URL: https://github.com/apache/doris/pull/13143#discussion_r995760274
##########
docker/thirdparties/start-thirdparties-docker.sh:
##########
@@ -20,8 +20,38 @@
# This script will restart all thirdparty containers
################################################################
-################################################################
-# restart elasticsearch containers
-################################################################
-docker compose -f docker-compose/elasticsearch.yaml down
-docker compose -f docker-compose/elasticsearch.yaml up -d
+set -eo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" &>/dev/null && pwd)"
+
+# If you want to start multi group of these containers on same host,
+# Change this to a specific string.
+# Do not use "_" or other sepcial characters, only number and alphabeta.
+# NOTICE: change this uid will modify the file in docker-compose.
+CONTAINER_UID="doris--"
+
+# elasticsearch
+# docker compose -f "${ROOT}"/docker-compose/elasticsearch/elasticsearch.yaml down
+# docker compose -f "${ROOT}"/docker-compose/elasticsearch/elasticsearch.yaml up -d
+
+# mysql 5.7
+sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml
+sudo docker compose -f "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml --env-file "${ROOT}"/docker-compose/mysql/mysql-5.7.env down
+sudo mkdir -p "${ROOT}"/docker-compose/mysql/data/
+sudo rm "${ROOT}"/docker-compose/mysql/data/* -rf
+sudo docker compose -f "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml --env-file "${ROOT}"/docker-compose/mysql/mysql-5.7.env up -d
+
+# pg 14
+sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml
+sudo docker compose -f "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml --env-file "${ROOT}"/docker-compose/postgresql/postgresql-14.env down
+sudo mkdir -p "${ROOT}"/docker-compose/postgresql/data/data
+sudo rm "${ROOT}"/docker-compose/postgresql/data/data/* -rf
+sudo docker compose -f "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml --env-file "${ROOT}"/docker-compose/postgresql/postgresql-14.env up -d
+
+# hive
+# before start it, you need to download parquet file package, see "README" in "docker-compose/hive/scripts/"
+sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/hive/hive-2x.yaml
+sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/hive/hadoop-hive.env.tpl
+sudo sh "${ROOT}"/docker-compose/hive/gen_env.sh
Review Comment:
this needs to be `bash` not `sh`. I don't know why, but this is the reality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13143: [test](jdbc) add jdbc regression test
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13143:
URL: https://github.com/apache/doris/pull/13143#issuecomment-1272472238
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/3213234459") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In docker/thirdparties/docker-compose/hive/gen_env.sh line 27:
cp ${ROOT}/hadoop-hive.env.tpl ${ROOT}/hadoop-hive.env
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
cp "${ROOT}"/hadoop-hive.env.tpl "${ROOT}"/hadoop-hive.env
In docker/thirdparties/docker-compose/hive/gen_env.sh line 30:
HOST_NAME=`hostname`
^--------^ SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
Did you mean:
HOST_NAME=$(hostname)
In docker/thirdparties/docker-compose/hive/gen_env.sh line 31:
echo "CORE_CONF_fs_defaultFS=hdfs://${HOST_NAME}:8020" >> ${ROOT}/hadoop-hive.env
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
echo "CORE_CONF_fs_defaultFS=hdfs://${HOST_NAME}:8020" >> "${ROOT}"/hadoop-hive.env
In docker/thirdparties/docker-compose/hive/gen_env.sh line 32:
echo "HOST_NAME=${HOST_NAME}" >> ${ROOT}/hadoop-hive.env
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
echo "HOST_NAME=${HOST_NAME}" >> "${ROOT}"/hadoop-hive.env
In docker/thirdparties/docker-compose/hive/scripts/healthy_check.sh line 19:
if [ ! -f "/mnt/SUCCESS" ]; then
^---------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
Did you mean:
if [[ ! -f "/mnt/SUCCESS" ]]; then
In docker/thirdparties/docker-compose/hive/scripts/hive-metastore.sh line 21:
if [ ! -d "/mnt/scripts/tpch1.db" ];then
^-- SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
Did you mean:
if [[ ! -d "/mnt/scripts/tpch1.db" ]];then
For more information:
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
https://www.shellcheck.net/wiki/SC2006 -- Use $(...) notation instead of le...
https://www.shellcheck.net/wiki/SC2292 -- Prefer [[ ]] over [ ] for tests i...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' returned error 1 finding the following formatting issues:
----------
--- docker/thirdparties/docker-compose/hive/gen_env.sh.orig
+++ docker/thirdparties/docker-compose/hive/gen_env.sh
@@ -27,6 +27,6 @@
cp ${ROOT}/hadoop-hive.env.tpl ${ROOT}/hadoop-hive.env
# Need to set hostname of container to same as host machine's.
# Otherwise, the doris process can not connect to namenode directly.
-HOST_NAME=`hostname`
-echo "CORE_CONF_fs_defaultFS=hdfs://${HOST_NAME}:8020" >> ${ROOT}/hadoop-hive.env
-echo "HOST_NAME=${HOST_NAME}" >> ${ROOT}/hadoop-hive.env
+HOST_NAME=$(hostname)
+echo "CORE_CONF_fs_defaultFS=hdfs://${HOST_NAME}:8020" >>${ROOT}/hadoop-hive.env
+echo "HOST_NAME=${HOST_NAME}" >>${ROOT}/hadoop-hive.env
--- docker/thirdparties/docker-compose/hive/scripts/healthy_check.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/healthy_check.sh
@@ -21,4 +21,3 @@
else
exit 0
fi
-
--- docker/thirdparties/docker-compose/hive/scripts/hive-metastore.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/hive-metastore.sh
@@ -18,7 +18,7 @@
nohup /opt/hive/bin/hive --service metastore &
sleep 10
-if [ ! -d "/mnt/scripts/tpch1.db" ];then
+if [ ! -d "/mnt/scripts/tpch1.db" ]; then
echo "/mnt/scripts/tpch1.db does not exist"
exit 1
fi
@@ -34,6 +34,6 @@
touch /mnt/SUCCESS
# Avoid container exit
-while true;do
+while true; do
sleep 1
done
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org