You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "LiBinfeng-01 (via GitHub)" <gi...@apache.org> on 2023/06/12 10:14:50 UTC
[GitHub] [doris] LiBinfeng-01 opened a new pull request, #20716: [Feature](Nereids) Add nereids ut
LiBinfeng-01 opened a new pull request, #20716:
URL: https://github.com/apache/doris/pull/20716
## Proposed changes
Two main changes:
1、add nereids ut
2、change minidump serialization of statistic messages and some interface between main logic of nereids optimizer and minidump
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1627955582
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5503192509") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In minidump/nereids_ut.sh line 172:
${JAVA}" ${final_java_opt} -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" ${OPT_VERSION:+${OPT_VERSION}} "$@" > "${temp_file}
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
^----^ SC2027 (warning): The surrounding quotes actually unquote this. Remove or escape them.
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-- SC2068 (error): Double quote array expansions to avoid re-splitting elements.
^-- SC2145 (error): Argument mixes string and array. Use * or separate argument.
^-- SC2027 (warning): The surrounding quotes actually unquote this. Remove or escape them.
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"${JAVA}"" ${final_java_opt} -Xmx2g org.apache.doris.nereids.minidump.Minidump ""${dir}"" ${OPT_VERSION:+${OPT_VERSION}} "$@" > ""${temp_file}"
For more information:
https://www.shellcheck.net/wiki/SC2068 -- Double quote array expansions to ...
https://www.shellcheck.net/wiki/SC2145 -- Argument mixes string and array. ...
https://www.shellcheck.net/wiki/SC2027 -- The surrounding quotes actually u...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' found no issues.
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on a diff in pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on code in PR #20716:
URL: https://github.com/apache/doris/pull/20716#discussion_r1268849217
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -549,24 +548,26 @@ private Statistics computeFilter(Filter filter) {
}
private ColumnStatistic getColumnStatistic(TableIf table, String colName) {
- if (totalColumnStatisticMap.get(table.getName() + colName) != null) {
- return totalColumnStatisticMap.get(table.getName() + colName);
- } else if (isPlayNereidsDump) {
- return ColumnStatistic.UNKNOWN;
- } else {
- long catalogId;
- long dbId;
- try {
- catalogId = table.getDatabase().getCatalog().getId();
- dbId = table.getDatabase().getId();
- } catch (Exception e) {
- // Use -1 for catalog id and db id when failed to get them from metadata.
- // This is OK because catalog id and db id is not in the hashcode function of ColumnStatistics cache
- // and the table id is globally unique.
- LOG.debug(String.format("Fail to get catalog id and db id for table %s", table.getName()));
- catalogId = -1;
- dbId = -1;
+ long catalogId;
+ long dbId;
+ try {
+ catalogId = table.getDatabase().getCatalog().getId();
+ dbId = table.getDatabase().getId();
+ } catch (Exception e) {
+ // Use -1 for catalog id and db id when failed to get them from metadata.
+ // This is OK because catalog id and db id is not in the hashcode function of ColumnStatistics cache
+ // and the table id is globally unique.
+ LOG.debug(String.format("Fail to get catalog id and db id for table %s", table.getName()));
+ catalogId = -1;
+ dbId = -1;
+ }
+ if (isPlayNereidsDump) {
Review Comment:
we do not have Env.getCurrentEnv() when replaying minidump, so it can not load to cache
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1631845337
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1645072554
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1631981885
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1644885585
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1633863017
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 52 seconds
stream load tsv: 511 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.8 seconds inserted 10000000 Rows, about 335K ops/s
storage size: 17168860436 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230713171102_clickbench_pr_177992.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1645696532
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on a diff in pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on code in PR #20716:
URL: https://github.com/apache/doris/pull/20716#discussion_r1268842799
##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -978,6 +984,9 @@ public void setMaxJoinNumBushyTree(int maxJoinNumBushyTree) {
@VariableMgr.VarAttr(name = ENABLE_FOLD_NONDETERMINISTIC_FN)
public boolean enableFoldNondeterministicFn = true;
+ @VariableMgr.VarAttr(name = MINIDUMP_PATH)
+ public String minidumpPath = "default";
Review Comment:
get
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20716: [Feature](Nereids) Add nereids ut
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1587028898
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5242317518") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In minidump/nereids_ut.sh line 41:
IMAGE_PATH=''
^--------^ SC2034 (warning): IMAGE_PATH appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 47:
RUN_DAEMON=1
^--------^ SC2034 (warning): RUN_DAEMON appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 55:
HELPER="$2"
^----^ SC2034 (warning): HELPER appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 59:
IMAGE_TOOL=1
^--------^ SC2034 (warning): IMAGE_TOOL appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 177:
if [[ -f "${pidfile}" ]]; then
^--------^ SC2154 (warning): pidfile is referenced but not assigned.
In minidump/nereids_ut.sh line 187:
LIMIT=/bin/limit
^---^ SC2034 (warning): LIMIT appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 195:
if [[ -d "$dir" ]]; then # Check if the path is a directory
^--^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
if [[ -d "${dir}" ]]; then # Check if the path is a directory
In minidump/nereids_ut.sh line 196:
query_id=$(echo "${dir: -33}")
^-------------------^ SC2116 (style): Useless echo? Instead of 'cmd $(echo foo)', just use 'cmd foo'.
In minidump/nereids_ut.sh line 198:
"${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump $dir ${OPT_VERSION:+${OPT_VERSION}} "$@" > "$temp_file"
^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
"${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" ${OPT_VERSION:+${OPT_VERSION}} "$@" > "${temp_file}"
In minidump/nereids_ut.sh line 200:
result=$(cat "$temp_file")
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
result=$(cat "${temp_file}")
In minidump/nereids_ut.sh line 201:
echo "$query_id_head $query_id $sql_head $result" |tee >> "${LOG_DIR}/minidump.out"
^------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
echo "${query_id_head} ${query_id} ${sql_head} ${result}" |tee >> "${LOG_DIR}/minidump.out"
For more information:
https://www.shellcheck.net/wiki/SC2034 -- HELPER appears unused. Verify use...
https://www.shellcheck.net/wiki/SC2154 -- pidfile is referenced but not ass...
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' returned error 1 finding the following formatting issues:
----------
--- minidump/nereids_ut.sh.orig
+++ minidump/nereids_ut.sh
@@ -192,15 +192,13 @@
sql_head="|Sql="
for dir in "${DUMP_FILE_DIR}"/*; do
- if [[ -d "$dir" ]]; then # Check if the path is a directory
- query_id=$(echo "${dir: -33}")
- temp_file=${LOG_DIR}/temp
- "${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump $dir ${OPT_VERSION:+${OPT_VERSION}} "$@" > "$temp_file"
+ if [[ -d "$dir" ]]; then # Check if the path is a directory
+ query_id=$(echo "${dir: -33}")
+ temp_file=${LOG_DIR}/temp
+ "${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump $dir ${OPT_VERSION:+${OPT_VERSION}} "$@" >"$temp_file"
- result=$(cat "$temp_file")
- echo "$query_id_head $query_id $sql_head $result" |tee >> "${LOG_DIR}/minidump.out"
+ result=$(cat "$temp_file")
+ echo "$query_id_head $query_id $sql_head $result" | tee >>"${LOG_DIR}/minidump.out"
fi
done
-date >> "${LOG_DIR}/minidump.out"
-
-
+date >>"${LOG_DIR}/minidump.out"
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #20716: [feature](Nereids) Add minidump replay and refactor user feature of minidump
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #20716:
URL: https://github.com/apache/doris/pull/20716
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1627951195
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5503150130") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In minidump/nereids_ut.sh line 39:
RUN_DAEMON=0
^--------^ SC2034 (warning): RUN_DAEMON appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 40:
HELPER=''
^----^ SC2034 (warning): HELPER appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 174:
${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" ${OPT_VERSION:+${OPT_VERSION}} "$@" > "${temp_file}
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
^----^ SC2027 (warning): The surrounding quotes actually unquote this. Remove or escape them.
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-- SC2068 (error): Double quote array expansions to avoid re-splitting elements.
^-- SC2145 (error): Argument mixes string and array. Use * or separate argument.
^-- SC2027 (warning): The surrounding quotes actually unquote this. Remove or escape them.
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"${JAVA}"" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump ""${dir}"" ${OPT_VERSION:+${OPT_VERSION}} "$@" > ""${temp_file}"
For more information:
https://www.shellcheck.net/wiki/SC2068 -- Double quote array expansions to ...
https://www.shellcheck.net/wiki/SC2145 -- Argument mixes string and array. ...
https://www.shellcheck.net/wiki/SC2027 -- The surrounding quotes actually u...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' returned error 1 finding the following formatting issues:
----------
--- minidump/nereids_ut.sh.orig
+++ minidump/nereids_ut.sh
@@ -168,14 +168,12 @@
sql_head="|Sql="
for dir in "${DUMP_FILE_DIR}"/*; do
- if [[ -d "${dir}" ]]; then # Check if the path is a directory
- query_id=${dir: -33}
- temp_file=${LOG_DIR}/temp
+ if [[ -d "${dir}" ]]; then # Check if the path is a directory
+ query_id=${dir: -33}
+ temp_file=${LOG_DIR}/temp
${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" ${OPT_VERSION:+${OPT_VERSION}} "$@" > "${temp_file}
- result=$(cat "${temp_file}")
- echo "${query_id_head} ${query_id} ${sql_head} ${result}" |tee >> "${LOG_DIR}/minidump.out"
+ result=$(cat "${temp_file}")
+ echo "${query_id_head} ${query_id} ${sql_head} ${result}" | tee >>"${LOG_DIR}/minidump.out"
fi
done
-date >> "${LOG_DIR}/minidump.out"
-
-
+date >>"${LOG_DIR}/minidump.out"
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1631735297
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5526721334") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In minidump/nereids_ut.sh line 172:
"${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" "${OPT_VERSION}" "$@" result=$(cat "${temp_file}") >"${temp_file}"
^-------------------^ SC2046 (warning): Quote this to prevent word splitting.
^------------^ SC2094 (info): Make sure not to read and write the same file in the same pipeline.
^------------^ SC2094 (info): Make sure not to read and write the same file in the same pipeline.
For more information:
https://www.shellcheck.net/wiki/SC2046 -- Quote this to prevent word splitt...
https://www.shellcheck.net/wiki/SC2094 -- Make sure not to read and write t...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' found no issues.
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1635361205
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 50.56 seconds
stream load tsv: 539 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17169290126 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1647297425
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on a diff in pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on code in PR #20716:
URL: https://github.com/apache/doris/pull/20716#discussion_r1268852004
##########
fe/fe-core/src/test/java/org/apache/doris/nereids/minidump/MinidumpUtTest.java:
##########
@@ -0,0 +1,43 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.minidump;
+
+import org.json.JSONObject;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+class MinidumpUtTest {
+
+ @Test
+ public void testMinidumpUt() {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1628628143
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5506697412") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In minidump/nereids_ut.sh line 172:
${JAVA} "${final_java_opt}" -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" "${OPT_VERSION}" "$@" > "${temp_file}" result=$(cat "${temp_file}")
^------------^ SC2094 (info): Make sure not to read and write the same file in the same pipeline.
^-------------------^ SC2046 (warning): Quote this to prevent word splitting.
^------------^ SC2094 (info): Make sure not to read and write the same file in the same pipeline.
For more information:
https://www.shellcheck.net/wiki/SC2046 -- Quote this to prevent word splitt...
https://www.shellcheck.net/wiki/SC2094 -- Make sure not to read and write t...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' returned error 1 finding the following formatting issues:
----------
--- minidump/nereids_ut.sh.orig
+++ minidump/nereids_ut.sh
@@ -169,7 +169,7 @@
if [[ -d "${dir}" ]]; then # Check if the path is a directory
query_id=${dir: -33}
temp_file=${LOG_DIR}/temp
- ${JAVA} "${final_java_opt}" -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" "${OPT_VERSION}" "$@" > "${temp_file}" result=$(cat "${temp_file}")
+ ${JAVA} "${final_java_opt}" -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" "${OPT_VERSION}" "$@" result=$(cat "${temp_file}") >"${temp_file}"
echo "${query_id_head} ${query_id} ${sql_head} ${result}" | tee >>"${LOG_DIR}/minidump.out"
fi
done
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1628428831
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5505508845") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In minidump/nereids_ut.sh line 172:
${JAVA}" ${final_java_opt} -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" ${OPT_VERSION:+${OPT_VERSION}} "$@" > "${temp_file}
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
^----^ SC2027 (warning): The surrounding quotes actually unquote this. Remove or escape them.
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-- SC2068 (error): Double quote array expansions to avoid re-splitting elements.
^-- SC2145 (error): Argument mixes string and array. Use * or separate argument.
^-- SC2027 (warning): The surrounding quotes actually unquote this. Remove or escape them.
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"${JAVA}"" ${final_java_opt} -Xmx2g org.apache.doris.nereids.minidump.Minidump ""${dir}"" ${OPT_VERSION:+${OPT_VERSION}} "$@" > ""${temp_file}"
For more information:
https://www.shellcheck.net/wiki/SC2068 -- Double quote array expansions to ...
https://www.shellcheck.net/wiki/SC2145 -- Argument mixes string and array. ...
https://www.shellcheck.net/wiki/SC2027 -- The surrounding quotes actually u...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' found no issues.
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1633805444
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1633807448
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1635321073
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1645784946
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.54 seconds
stream load tsv: 544 seconds loaded 74807831229 Bytes, about 131 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162330455 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1630497777
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5518339933") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In minidump/nereids_ut.sh line 172:
${JAVA} "${final_java_opt}" -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" "${OPT_VERSION}" "$@" > "${temp_file}" result=$(cat "${temp_file}")
^------------^ SC2094 (info): Make sure not to read and write the same file in the same pipeline.
^-------------------^ SC2046 (warning): Quote this to prevent word splitting.
^------------^ SC2094 (info): Make sure not to read and write the same file in the same pipeline.
For more information:
https://www.shellcheck.net/wiki/SC2046 -- Quote this to prevent word splitt...
https://www.shellcheck.net/wiki/SC2094 -- Make sure not to read and write t...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' returned error 1 finding the following formatting issues:
----------
--- minidump/nereids_ut.sh.orig
+++ minidump/nereids_ut.sh
@@ -169,7 +169,7 @@
if [[ -d "${dir}" ]]; then # Check if the path is a directory
query_id=${dir: -33}
temp_file=${LOG_DIR}/temp
- ${JAVA} "${final_java_opt}" -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" "${OPT_VERSION}" "$@" > "${temp_file}" result=$(cat "${temp_file}")
+ ${JAVA} "${final_java_opt}" -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" "${OPT_VERSION}" "$@" result=$(cat "${temp_file}") >"${temp_file}"
echo "${query_id_head} ${query_id} ${sql_head} ${result}" | tee >>"${LOG_DIR}/minidump.out"
fi
done
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1630504822
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5518366132") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In minidump/nereids_ut.sh line 172:
${JAVA} "${final_java_opt}" -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" "${OPT_VERSION}" "$@" result=$(cat "${temp_file}") >"${temp_file}"
^-------------------^ SC2046 (warning): Quote this to prevent word splitting.
^------------^ SC2094 (info): Make sure not to read and write the same file in the same pipeline.
^------------^ SC2094 (info): Make sure not to read and write the same file in the same pipeline.
For more information:
https://www.shellcheck.net/wiki/SC2046 -- Quote this to prevent word splitt...
https://www.shellcheck.net/wiki/SC2094 -- Make sure not to read and write t...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' found no issues.
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1633507542
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 51.1 seconds
stream load tsv: 507 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17167978017 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230713115442_clickbench_pr_177777.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #20716:
URL: https://github.com/apache/doris/pull/20716#discussion_r1268314793
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java:
##########
@@ -65,7 +65,16 @@ class CostModelV1 extends PlanVisitor<Cost, PlanContext> {
// the penalty factor is no more than BROADCAST_JOIN_SKEW_PENALTY_LIMIT
static final double BROADCAST_JOIN_SKEW_RATIO = 30.0;
static final double BROADCAST_JOIN_SKEW_PENALTY_LIMIT = 2.0;
- private int beNumber = Math.max(1, ConnectContext.get().getEnv().getClusterInfo().getBackendsNumber(true));
+ private int beNumber = 1;
+
+ public CostModelV1() {
+ if (ConnectContext.get().getSessionVariable().isPlayNereidsDump()) {
+ beNumber = ConnectContext.get().getSessionVariable().getBeNumber();
+ } else {
+ beNumber = Math.max(1, ConnectContext.get().getEnv().getClusterInfo().getBackendsNumber(true));
+ ConnectContext.get().getSessionVariable().setBeNumber(beNumber);
Review Comment:
should not change session variable anytime
##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -978,6 +984,9 @@ public void setMaxJoinNumBushyTree(int maxJoinNumBushyTree) {
@VariableMgr.VarAttr(name = ENABLE_FOLD_NONDETERMINISTIC_FN)
public boolean enableFoldNondeterministicFn = true;
+ @VariableMgr.VarAttr(name = MINIDUMP_PATH)
+ public String minidumpPath = "default";
Review Comment:
nereids_minidump_path, btw use empty string to present default maybe better
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java:
##########
@@ -136,6 +136,9 @@ private LogicalPlan bindWithCurrentDb(CascadesContext cascadesContext, UnboundRe
TableIf table = null;
if (cascadesContext.getTables() != null) {
table = cascadesContext.getTableByName(tableName);
+ if (ConnectContext.get().getSessionVariable().isPlayNereidsDump() && table == null) {
+ throw new AnalysisException("Can not find table:" + tableName);
+ }
Review Comment:
if getTableByName only used for minidump, please throw exception in it self and change the method name to reflect it used for minidump
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -549,24 +548,26 @@ private Statistics computeFilter(Filter filter) {
}
private ColumnStatistic getColumnStatistic(TableIf table, String colName) {
- if (totalColumnStatisticMap.get(table.getName() + colName) != null) {
- return totalColumnStatisticMap.get(table.getName() + colName);
- } else if (isPlayNereidsDump) {
- return ColumnStatistic.UNKNOWN;
- } else {
- long catalogId;
- long dbId;
- try {
- catalogId = table.getDatabase().getCatalog().getId();
- dbId = table.getDatabase().getId();
- } catch (Exception e) {
- // Use -1 for catalog id and db id when failed to get them from metadata.
- // This is OK because catalog id and db id is not in the hashcode function of ColumnStatistics cache
- // and the table id is globally unique.
- LOG.debug(String.format("Fail to get catalog id and db id for table %s", table.getName()));
- catalogId = -1;
- dbId = -1;
+ long catalogId;
+ long dbId;
+ try {
+ catalogId = table.getDatabase().getCatalog().getId();
+ dbId = table.getDatabase().getId();
+ } catch (Exception e) {
+ // Use -1 for catalog id and db id when failed to get them from metadata.
+ // This is OK because catalog id and db id is not in the hashcode function of ColumnStatistics cache
+ // and the table id is globally unique.
+ LOG.debug(String.format("Fail to get catalog id and db id for table %s", table.getName()));
+ catalogId = -1;
+ dbId = -1;
+ }
+ if (isPlayNereidsDump) {
Review Comment:
i think the better way to do this is put all column stats in dump file into cache first. and then we don't need to change any of the code here
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/rewrite/TopicRewriteJob.java:
##########
@@ -51,4 +51,9 @@ public void execute(JobContext jobContext) {
public boolean isOnce() {
return true;
}
+
+ @Override
+ public String getRuleType() {
Review Comment:
why add this interface? u do not use it in anyplace
##########
minidump/nereids_ut.sh:
##########
Review Comment:
add readme
##########
fe/fe-core/src/test/java/org/apache/doris/nereids/minidump/MinidumpUtTest.java:
##########
@@ -0,0 +1,43 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.minidump;
+
+import org.json.JSONObject;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+class MinidumpUtTest {
+
+ @Test
+ public void testMinidumpUt() {
Review Comment:
add comment to explain how to use it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1645738958
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1619352598
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5450041207") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In minidump/nereids_ut.sh line 41:
IMAGE_PATH=''
^--------^ SC2034 (warning): IMAGE_PATH appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 47:
RUN_DAEMON=1
^--------^ SC2034 (warning): RUN_DAEMON appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 55:
HELPER="$2"
^----^ SC2034 (warning): HELPER appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 59:
IMAGE_TOOL=1
^--------^ SC2034 (warning): IMAGE_TOOL appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 177:
if [[ -f "${pidfile}" ]]; then
^--------^ SC2154 (warning): pidfile is referenced but not assigned.
In minidump/nereids_ut.sh line 187:
LIMIT=/bin/limit
^---^ SC2034 (warning): LIMIT appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 195:
if [[ -d "$dir" ]]; then # Check if the path is a directory
^--^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
if [[ -d "${dir}" ]]; then # Check if the path is a directory
In minidump/nereids_ut.sh line 196:
query_id=$(echo "${dir: -33}")
^-------------------^ SC2116 (style): Useless echo? Instead of 'cmd $(echo foo)', just use 'cmd foo'.
In minidump/nereids_ut.sh line 198:
"${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump $dir ${OPT_VERSION:+${OPT_VERSION}} "$@" > "$temp_file"
^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
"${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" ${OPT_VERSION:+${OPT_VERSION}} "$@" > "${temp_file}"
In minidump/nereids_ut.sh line 200:
result=$(cat "$temp_file")
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
result=$(cat "${temp_file}")
In minidump/nereids_ut.sh line 201:
echo "$query_id_head $query_id $sql_head $result" |tee >> "${LOG_DIR}/minidump.out"
^------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
echo "${query_id_head} ${query_id} ${sql_head} ${result}" |tee >> "${LOG_DIR}/minidump.out"
For more information:
https://www.shellcheck.net/wiki/SC2034 -- HELPER appears unused. Verify use...
https://www.shellcheck.net/wiki/SC2154 -- pidfile is referenced but not ass...
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' returned error 1 finding the following formatting issues:
----------
--- minidump/nereids_ut.sh.orig
+++ minidump/nereids_ut.sh
@@ -192,15 +192,13 @@
sql_head="|Sql="
for dir in "${DUMP_FILE_DIR}"/*; do
- if [[ -d "$dir" ]]; then # Check if the path is a directory
- query_id=$(echo "${dir: -33}")
- temp_file=${LOG_DIR}/temp
- "${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump $dir ${OPT_VERSION:+${OPT_VERSION}} "$@" > "$temp_file"
+ if [[ -d "$dir" ]]; then # Check if the path is a directory
+ query_id=$(echo "${dir: -33}")
+ temp_file=${LOG_DIR}/temp
+ "${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump $dir ${OPT_VERSION:+${OPT_VERSION}} "$@" >"$temp_file"
- result=$(cat "$temp_file")
- echo "$query_id_head $query_id $sql_head $result" |tee >> "${LOG_DIR}/minidump.out"
+ result=$(cat "$temp_file")
+ echo "$query_id_head $query_id $sql_head $result" | tee >>"${LOG_DIR}/minidump.out"
fi
done
-date >> "${LOG_DIR}/minidump.out"
-
-
+date >>"${LOG_DIR}/minidump.out"
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1598391558
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5320593201") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In minidump/nereids_ut.sh line 41:
IMAGE_PATH=''
^--------^ SC2034 (warning): IMAGE_PATH appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 47:
RUN_DAEMON=1
^--------^ SC2034 (warning): RUN_DAEMON appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 55:
HELPER="$2"
^----^ SC2034 (warning): HELPER appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 59:
IMAGE_TOOL=1
^--------^ SC2034 (warning): IMAGE_TOOL appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 177:
if [[ -f "${pidfile}" ]]; then
^--------^ SC2154 (warning): pidfile is referenced but not assigned.
In minidump/nereids_ut.sh line 187:
LIMIT=/bin/limit
^---^ SC2034 (warning): LIMIT appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 195:
if [[ -d "$dir" ]]; then # Check if the path is a directory
^--^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
if [[ -d "${dir}" ]]; then # Check if the path is a directory
In minidump/nereids_ut.sh line 196:
query_id=$(echo "${dir: -33}")
^-------------------^ SC2116 (style): Useless echo? Instead of 'cmd $(echo foo)', just use 'cmd foo'.
In minidump/nereids_ut.sh line 198:
"${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump $dir ${OPT_VERSION:+${OPT_VERSION}} "$@" > "$temp_file"
^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
"${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" ${OPT_VERSION:+${OPT_VERSION}} "$@" > "${temp_file}"
In minidump/nereids_ut.sh line 200:
result=$(cat "$temp_file")
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
result=$(cat "${temp_file}")
In minidump/nereids_ut.sh line 201:
echo "$query_id_head $query_id $sql_head $result" |tee >> "${LOG_DIR}/minidump.out"
^------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
echo "${query_id_head} ${query_id} ${sql_head} ${result}" |tee >> "${LOG_DIR}/minidump.out"
For more information:
https://www.shellcheck.net/wiki/SC2034 -- HELPER appears unused. Verify use...
https://www.shellcheck.net/wiki/SC2154 -- pidfile is referenced but not ass...
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' returned error 1 finding the following formatting issues:
----------
--- minidump/nereids_ut.sh.orig
+++ minidump/nereids_ut.sh
@@ -192,15 +192,13 @@
sql_head="|Sql="
for dir in "${DUMP_FILE_DIR}"/*; do
- if [[ -d "$dir" ]]; then # Check if the path is a directory
- query_id=$(echo "${dir: -33}")
- temp_file=${LOG_DIR}/temp
- "${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump $dir ${OPT_VERSION:+${OPT_VERSION}} "$@" > "$temp_file"
+ if [[ -d "$dir" ]]; then # Check if the path is a directory
+ query_id=$(echo "${dir: -33}")
+ temp_file=${LOG_DIR}/temp
+ "${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump $dir ${OPT_VERSION:+${OPT_VERSION}} "$@" >"$temp_file"
- result=$(cat "$temp_file")
- echo "$query_id_head $query_id $sql_head $result" |tee >> "${LOG_DIR}/minidump.out"
+ result=$(cat "$temp_file")
+ echo "$query_id_head $query_id $sql_head $result" | tee >>"${LOG_DIR}/minidump.out"
fi
done
-date >> "${LOG_DIR}/minidump.out"
-
-
+date >>"${LOG_DIR}/minidump.out"
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1621653093
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5464208799") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In minidump/nereids_ut.sh line 41:
IMAGE_PATH=''
^--------^ SC2034 (warning): IMAGE_PATH appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 47:
RUN_DAEMON=1
^--------^ SC2034 (warning): RUN_DAEMON appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 55:
HELPER="$2"
^----^ SC2034 (warning): HELPER appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 59:
IMAGE_TOOL=1
^--------^ SC2034 (warning): IMAGE_TOOL appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 177:
if [[ -f "${pidfile}" ]]; then
^--------^ SC2154 (warning): pidfile is referenced but not assigned.
In minidump/nereids_ut.sh line 187:
LIMIT=/bin/limit
^---^ SC2034 (warning): LIMIT appears unused. Verify use (or export if used externally).
In minidump/nereids_ut.sh line 195:
if [[ -d "$dir" ]]; then # Check if the path is a directory
^--^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
if [[ -d "${dir}" ]]; then # Check if the path is a directory
In minidump/nereids_ut.sh line 196:
query_id=$(echo "${dir: -33}")
^-------------------^ SC2116 (style): Useless echo? Instead of 'cmd $(echo foo)', just use 'cmd foo'.
In minidump/nereids_ut.sh line 198:
"${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump $dir ${OPT_VERSION:+${OPT_VERSION}} "$@" > "$temp_file"
^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
"${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" ${OPT_VERSION:+${OPT_VERSION}} "$@" > "${temp_file}"
In minidump/nereids_ut.sh line 200:
result=$(cat "$temp_file")
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
result=$(cat "${temp_file}")
In minidump/nereids_ut.sh line 201:
echo "$query_id_head $query_id $sql_head $result" |tee >> "${LOG_DIR}/minidump.out"
^------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
echo "${query_id_head} ${query_id} ${sql_head} ${result}" |tee >> "${LOG_DIR}/minidump.out"
For more information:
https://www.shellcheck.net/wiki/SC2034 -- HELPER appears unused. Verify use...
https://www.shellcheck.net/wiki/SC2154 -- pidfile is referenced but not ass...
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' returned error 1 finding the following formatting issues:
----------
--- minidump/nereids_ut.sh.orig
+++ minidump/nereids_ut.sh
@@ -192,15 +192,13 @@
sql_head="|Sql="
for dir in "${DUMP_FILE_DIR}"/*; do
- if [[ -d "$dir" ]]; then # Check if the path is a directory
- query_id=$(echo "${dir: -33}")
- temp_file=${LOG_DIR}/temp
- "${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump $dir ${OPT_VERSION:+${OPT_VERSION}} "$@" > "$temp_file"
+ if [[ -d "$dir" ]]; then # Check if the path is a directory
+ query_id=$(echo "${dir: -33}")
+ temp_file=${LOG_DIR}/temp
+ "${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump $dir ${OPT_VERSION:+${OPT_VERSION}} "$@" >"$temp_file"
- result=$(cat "$temp_file")
- echo "$query_id_head $query_id $sql_head $result" |tee >> "${LOG_DIR}/minidump.out"
+ result=$(cat "$temp_file")
+ echo "$query_id_head $query_id $sql_head $result" | tee >>"${LOG_DIR}/minidump.out"
fi
done
-date >> "${LOG_DIR}/minidump.out"
-
-
+date >>"${LOG_DIR}/minidump.out"
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1632004445
run clickbench
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1643618098
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1645107149
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.61 seconds
stream load tsv: 507 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17166910626 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on a diff in pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on code in PR #20716:
URL: https://github.com/apache/doris/pull/20716#discussion_r1268870852
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java:
##########
@@ -65,7 +65,16 @@ class CostModelV1 extends PlanVisitor<Cost, PlanContext> {
// the penalty factor is no more than BROADCAST_JOIN_SKEW_PENALTY_LIMIT
static final double BROADCAST_JOIN_SKEW_RATIO = 30.0;
static final double BROADCAST_JOIN_SKEW_PENALTY_LIMIT = 2.0;
- private int beNumber = Math.max(1, ConnectContext.get().getEnv().getClusterInfo().getBackendsNumber(true));
+ private int beNumber = 1;
+
+ public CostModelV1() {
+ if (ConnectContext.get().getSessionVariable().isPlayNereidsDump()) {
+ beNumber = ConnectContext.get().getSessionVariable().getBeNumber();
+ } else {
+ beNumber = Math.max(1, ConnectContext.get().getEnv().getClusterInfo().getBackendsNumber(true));
+ ConnectContext.get().getSessionVariable().setBeNumber(beNumber);
Review Comment:
get
##########
minidump/nereids_ut.sh:
##########
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1647297460
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on a diff in pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on code in PR #20716:
URL: https://github.com/apache/doris/pull/20716#discussion_r1268845053
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java:
##########
@@ -136,6 +136,9 @@ private LogicalPlan bindWithCurrentDb(CascadesContext cascadesContext, UnboundRe
TableIf table = null;
if (cascadesContext.getTables() != null) {
table = cascadesContext.getTableByName(tableName);
+ if (ConnectContext.get().getSessionVariable().isPlayNereidsDump() && table == null) {
+ throw new AnalysisException("Can not find table:" + tableName);
+ }
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on a diff in pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on code in PR #20716:
URL: https://github.com/apache/doris/pull/20716#discussion_r1268846913
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/rewrite/TopicRewriteJob.java:
##########
@@ -51,4 +51,9 @@ public void execute(JobContext jobContext) {
public boolean isOnce() {
return true;
}
+
+ @Override
+ public String getRuleType() {
Review Comment:
used for trace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1643666885
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.03 seconds
stream load tsv: 510 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17162449605 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1631841338
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5527660871") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' found no issues.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' returned error 1 finding the following formatting issues:
----------
--- minidump/nereids_ut.sh.orig
+++ minidump/nereids_ut.sh
@@ -162,15 +162,15 @@
CLASSPATH="${DORIS_FE_JAR}:${CLASSPATH}"
export CLASSPATH="${CLASSPATH}:${DORIS_HOME}/lib:${DORIS_HOME}/conf"
-date >> "${LOG_DIR}/minidump.out"
+date >>"${LOG_DIR}/minidump.out"
query_id_head="|QueryId="
sql_head="|Sql="
# shellcheck disable=SC2045
for path in $(ls "${DUMP_FILE_DIR}"); do
query_id=${path:0:33}
temp_file=${LOG_DIR}/temp
- "${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump "${DUMP_FILE_DIR}/${path}" "${OPT_VERSION}" "$@" > "${temp_file}"
+ "${JAVA}" ${final_java_opt:+${final_java_opt}} -Xmx2g org.apache.doris.nereids.minidump.Minidump "${DUMP_FILE_DIR}/${path}" "${OPT_VERSION}" "$@" >"${temp_file}"
result=$(cat "${temp_file}")
- echo "${query_id_head} ${query_id} ${sql_head} ${result}" | tee >> "${LOG_DIR}/minidump.out"
+ echo "${query_id_head} ${query_id} ${sql_head} ${result}" | tee >>"${LOG_DIR}/minidump.out"
done
date >>"${LOG_DIR}/minidump.out"
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1631936443
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1633489633
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1633865759
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 51.65 seconds
stream load tsv: 539 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.8 seconds inserted 10000000 Rows, about 335K ops/s
storage size: 17164932270 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230713171249_clickbench_pr_177998.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20716: [Feature](Nereids) Add nereids unit test and refactor user feature of minidump
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20716:
URL: https://github.com/apache/doris/pull/20716#issuecomment-1628443671
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5505596267") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In minidump/nereids_ut.sh line 172:
${JAVA} ${final_java_opt} -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" ${OPT_VERSION} "$@" > ${temp_file}
^---------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
${JAVA} "${final_java_opt}" -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" "${OPT_VERSION}" "$@" > "${temp_file}"
For more information:
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
https://www.shellcheck.net/wiki/SC2248 -- Prefer double quoting even when v...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' returned error 1 finding the following formatting issues:
----------
--- minidump/nereids_ut.sh.orig
+++ minidump/nereids_ut.sh
@@ -169,7 +169,7 @@
if [[ -d "${dir}" ]]; then # Check if the path is a directory
query_id=${dir: -33}
temp_file=${LOG_DIR}/temp
- ${JAVA} ${final_java_opt} -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" ${OPT_VERSION} "$@" > ${temp_file}
+ ${JAVA} ${final_java_opt} -Xmx2g org.apache.doris.nereids.minidump.Minidump "${dir}" ${OPT_VERSION} "$@" >${temp_file}
result=$(cat "${temp_file}")
echo "${query_id_head} ${query_id} ${sql_head} ${result}" | tee >>"${LOG_DIR}/minidump.out"
fi
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org