You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xzj7019 (via GitHub)" <gi...@apache.org> on 2023/07/03 01:47:11 UTC
[GitHub] [doris] xzj7019 opened a new pull request, #21421: [improvement](nereids) Refine tpcds tools
xzj7019 opened a new pull request, #21421:
URL: https://github.com/apache/doris/pull/21421
## Proposed changes
Refine tpcds test tools, including split 99 cases into separate files, and refine 100g schema with range partition format.
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21421: [improvement](nereids) Refine tpcds tools
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21421:
URL: https://github.com/apache/doris/pull/21421#issuecomment-1617230943
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21421: [improvement](nereids) Refine tpcds tools
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21421:
URL: https://github.com/apache/doris/pull/21421#issuecomment-1617230868
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21421: [improvement](nereids) Refine tpcds tools
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21421:
URL: https://github.com/apache/doris/pull/21421#issuecomment-1617102957
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5439719427") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In tools/tpcds-tools/bin/create-tpcds-tables.sh line 80:
if [[ ${SCALE_FACTOR} -ne 1 ]] && [[ ${SCALE_FACTOR} -ne 100]]; then
^-- SC1009 (info): The mentioned syntax error was in this if expression.
^-- SC1073 (error): Couldn't parse this test expression. Fix to allow more checks.
^-- SC1020 (error): You need a space before the ]].
^-- SC1072 (error): Missing space before ]. Fix any mentioned problems and try again.
In tools/tpcds-tools/bin/run-tpcds-queries.sh line 135:
mysql -h${FE_HOST} -u${USER} -P${FE_QUERY_PORT} < ${TPCDS_QUERIES_DIR}/query${i}.sql > ${RESULT_DIR}/result${i}.out 2>${RESULT_DIR}/result${i}.log
^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^------------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mysql -h"${FE_HOST}" -u"${USER}" -P"${FE_QUERY_PORT}" < "${TPCDS_QUERIES_DIR}"/query"${i}".sql > "${RESULT_DIR}"/result"${i}".out 2>"${RESULT_DIR}"/result"${i}".log
In tools/tpcds-tools/bin/run-tpcds-queries.sh line 141:
mysql -h${FE_HOST} -u${USER} -P${FE_QUERY_PORT} < ${TPCDS_QUERIES_DIR}/query${i}.sql > ${RESULT_DIR}/result${i}.out 2>${RESULT_DIR}/result${i}.log
^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^------------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mysql -h"${FE_HOST}" -u"${USER}" -P"${FE_QUERY_PORT}" < "${TPCDS_QUERIES_DIR}"/query"${i}".sql > "${RESULT_DIR}"/result"${i}".out 2>"${RESULT_DIR}"/result"${i}".log
In tools/tpcds-tools/bin/run-tpcds-queries.sh line 147:
mysql -h${FE_HOST} -u${USER} -P${FE_QUERY_PORT} < ${TPCDS_QUERIES_DIR}/query${i}.sql > ${RESULT_DIR}/result${i}.out 2>${RESULT_DIR}/result${i}.log
^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^------------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
mysql -h"${FE_HOST}" -u"${USER}" -P"${FE_QUERY_PORT}" < "${TPCDS_QUERIES_DIR}"/query"${i}".sql > "${RESULT_DIR}"/result"${i}".out 2>"${RESULT_DIR}"/result"${i}".log
In tools/tpcds-tools/bin/run-tpcds-queries.sh line 168:
find "${RESULT_DIR}" -name "*.log" -type f -size 0c | xargs -n 1 rm -f
^-- SC2038 (warning): Use -print0/-0 or -exec + to allow for non-alphanumeric filenames.
In tools/tpcds-tools/bin/run-tpcds-queries.sh line 170:
ls ${RESULT_DIR}/*.log
^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
ls "${RESULT_DIR}"/*.log
For more information:
https://www.shellcheck.net/wiki/SC1020 -- You need a space before the ]].
https://www.shellcheck.net/wiki/SC2038 -- Use -print0/-0 or -exec + to allo...
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' returned error 1 finding the following formatting issues:
----------
tools/tpcds-tools/bin/create-tpcds-tables.sh:80:63: not a valid test operator: ;
--- tools/tpcds-tools/bin/run-tpcds-queries.sh.orig
+++ tools/tpcds-tools/bin/run-tpcds-queries.sh
@@ -59,9 +59,9 @@
shift
;;
-s)
- SCALE_FACTOR=$2
- shift 2
- ;;
+ SCALE_FACTOR=$2
+ shift 2
+ ;;
--)
shift
break
@@ -78,11 +78,11 @@
fi
if [[ ${SCALE_FACTOR} -eq 1 ]]; then
- echo "Running tpcds sf 1 queries"
- TPCDS_QUERIES_DIR="${CURDIR}/../queries_1"
+ echo "Running tpcds sf 1 queries"
+ TPCDS_QUERIES_DIR="${CURDIR}/../queries_1"
elif [[ ${SCALE_FACTOR} -eq 100 ]]; then
- echo "Running tpcds sf 100 queries"
- TPCDS_QUERIES_DIR="${CURDIR}/../queries_100"
+ echo "Running tpcds sf 100 queries"
+ TPCDS_QUERIES_DIR="${CURDIR}/../queries_100"
fi
check_prerequest() {
@@ -132,19 +132,19 @@
hot2=0
echo -ne "query${i}\t" | tee -a result.csv
start=$(date +%s%3N)
- mysql -h${FE_HOST} -u${USER} -P${FE_QUERY_PORT} < ${TPCDS_QUERIES_DIR}/query${i}.sql > ${RESULT_DIR}/result${i}.out 2>${RESULT_DIR}/result${i}.log
+ mysql -h${FE_HOST} -u${USER} -P${FE_QUERY_PORT} <${TPCDS_QUERIES_DIR}/query${i}.sql >${RESULT_DIR}/result${i}.out 2>${RESULT_DIR}/result${i}.log
end=$(date +%s%3N)
cold=$((end - start))
echo -ne "${cold}\t" | tee -a result.csv
start=$(date +%s%3N)
- mysql -h${FE_HOST} -u${USER} -P${FE_QUERY_PORT} < ${TPCDS_QUERIES_DIR}/query${i}.sql > ${RESULT_DIR}/result${i}.out 2>${RESULT_DIR}/result${i}.log
+ mysql -h${FE_HOST} -u${USER} -P${FE_QUERY_PORT} <${TPCDS_QUERIES_DIR}/query${i}.sql >${RESULT_DIR}/result${i}.out 2>${RESULT_DIR}/result${i}.log
end=$(date +%s%3N)
hot1=$((end - start))
echo -ne "${hot1}\t" | tee -a result.csv
start=$(date +%s%3N)
- mysql -h${FE_HOST} -u${USER} -P${FE_QUERY_PORT} < ${TPCDS_QUERIES_DIR}/query${i}.sql > ${RESULT_DIR}/result${i}.out 2>${RESULT_DIR}/result${i}.log
+ mysql -h${FE_HOST} -u${USER} -P${FE_QUERY_PORT} <${TPCDS_QUERIES_DIR}/query${i}.sql >${RESULT_DIR}/result${i}.out 2>${RESULT_DIR}/result${i}.log
end=$(date +%s%3N)
hot2=$((end - start))
echo -ne "${hot2}\t" | tee -a result.csv
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xzj7019 commented on pull request #21421: [improvement](nereids) Refine tpcds tools
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #21421:
URL: https://github.com/apache/doris/pull/21421#issuecomment-1617779536
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #21421: [improvement](nereids) Refine tpcds tools
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #21421:
URL: https://github.com/apache/doris/pull/21421
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21421: [improvement](nereids) Refine tpcds tools
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21421:
URL: https://github.com/apache/doris/pull/21421#issuecomment-1617161737
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5440202402") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In tools/tpcds-tools/bin/run-tpcds-queries.sh line 106:
echo "FE_HOST: ${FE_HOST}"
^--------^ SC2154 (warning): FE_HOST is referenced but not assigned.
In tools/tpcds-tools/bin/run-tpcds-queries.sh line 107:
echo "FE_QUERY_PORT: ${FE_QUERY_PORT}"
^--------------^ SC2154 (warning): FE_QUERY_PORT is referenced but not assigned.
In tools/tpcds-tools/bin/run-tpcds-queries.sh line 109:
echo "DB: ${DB}"
^---^ SC2154 (warning): DB is referenced but not assigned.
For more information:
https://www.shellcheck.net/wiki/SC2154 -- DB is referenced but not assigned.
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' returned error 1 finding the following formatting issues:
----------
--- tools/tpcds-tools/bin/create-tpcds-tables.sh.orig
+++ tools/tpcds-tools/bin/create-tpcds-tables.sh
@@ -59,9 +59,9 @@
shift
;;
-s)
- SCALE_FACTOR=$2
- shift 2
- ;;
+ SCALE_FACTOR=$2
+ shift 2
+ ;;
--)
shift
break
@@ -110,7 +110,7 @@
elif [[ ${SCALE_FACTOR} -eq 100 ]]; then
echo "Run SQLs from ${CURDIR}/../ddl/create-tpcds-tables-sf100.sql"
mysql -h"${FE_HOST}" -u"${USER}" -P"${FE_QUERY_PORT}" -D"${DB}" <"${CURDIR}"/../ddl/create-tpcds-tables-sf100.sql
-else
+else
echo "${SCALE_FACTOR} scale is NOT supported currently"
fi
--- tools/tpcds-tools/bin/run-tpcds-queries.sh.orig
+++ tools/tpcds-tools/bin/run-tpcds-queries.sh
@@ -59,9 +59,9 @@
shift
;;
-s)
- SCALE_FACTOR=$2
- shift 2
- ;;
+ SCALE_FACTOR=$2
+ shift 2
+ ;;
--)
shift
break
@@ -78,14 +78,14 @@
fi
if [[ ${SCALE_FACTOR} -eq 1 ]]; then
- echo "Running tpcds sf 1 queries"
- TPCDS_QUERIES_DIR="${CURDIR}/../queries_1"
+ echo "Running tpcds sf 1 queries"
+ TPCDS_QUERIES_DIR="${CURDIR}/../queries_1"
elif [[ ${SCALE_FACTOR} -eq 100 ]]; then
- echo "Running tpcds sf 100 queries"
- TPCDS_QUERIES_DIR="${CURDIR}/../queries_100"
+ echo "Running tpcds sf 100 queries"
+ TPCDS_QUERIES_DIR="${CURDIR}/../queries_100"
else
- echo "${SCALE_FACTOR} scale is NOT support currently."
- exit 1
+ echo "${SCALE_FACTOR} scale is NOT support currently."
+ exit 1
fi
check_prerequest() {
@@ -137,19 +137,19 @@
hot2=0
echo -ne "query${i}\t" | tee -a result.csv
start=$(date +%s%3N)
- mysql -h"${FE_HOST}" -u"${USER}" -P"${FE_QUERY_PORT}" -D"${DB}" --comments < "${TPCDS_QUERIES_DIR}"/query"${i}".sql > /dev/null
+ mysql -h"${FE_HOST}" -u"${USER}" -P"${FE_QUERY_PORT}" -D"${DB}" --comments <"${TPCDS_QUERIES_DIR}"/query"${i}".sql >/dev/null
end=$(date +%s%3N)
cold=$((end - start))
echo -ne "${cold}\t" | tee -a result.csv
start=$(date +%s%3N)
- mysql -h"${FE_HOST}" -u"${USER}" -P"${FE_QUERY_PORT}" -D"${DB}" --comments < "${TPCDS_QUERIES_DIR}"/query"${i}".sql > /dev/null
+ mysql -h"${FE_HOST}" -u"${USER}" -P"${FE_QUERY_PORT}" -D"${DB}" --comments <"${TPCDS_QUERIES_DIR}"/query"${i}".sql >/dev/null
end=$(date +%s%3N)
hot1=$((end - start))
echo -ne "${hot1}\t" | tee -a result.csv
start=$(date +%s%3N)
- mysql -h"${FE_HOST}" -u"${USER}" -P"${FE_QUERY_PORT}" -D"${DB}" --comments < "${TPCDS_QUERIES_DIR}"/query"${i}".sql > /dev/null
+ mysql -h"${FE_HOST}" -u"${USER}" -P"${FE_QUERY_PORT}" -D"${DB}" --comments <"${TPCDS_QUERIES_DIR}"/query"${i}".sql >/dev/null
end=$(date +%s%3N)
hot2=$((end - start))
echo -ne "${hot2}\t" | tee -a result.csv
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21421: [improvement](nereids) Refine tpcds tools
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21421:
URL: https://github.com/apache/doris/pull/21421#issuecomment-1617167202
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5440237290") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In tools/tpcds-tools/bin/run-tpcds-queries.sh line 106:
echo "FE_HOST: ${FE_HOST}"
^--------^ SC2154 (warning): FE_HOST is referenced but not assigned.
In tools/tpcds-tools/bin/run-tpcds-queries.sh line 107:
echo "FE_QUERY_PORT: ${FE_QUERY_PORT}"
^--------------^ SC2154 (warning): FE_QUERY_PORT is referenced but not assigned.
In tools/tpcds-tools/bin/run-tpcds-queries.sh line 109:
echo "DB: ${DB}"
^---^ SC2154 (warning): DB is referenced but not assigned.
For more information:
https://www.shellcheck.net/wiki/SC2154 -- DB is referenced but not assigned.
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' found no issues.
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org