You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "kaka11chen (via GitHub)" <gi...@apache.org> on 2023/06/21 08:21:20 UTC
[GitHub] [doris] kaka11chen opened a new pull request, #21074: [Feature](multi-catalog) Add hdfs benchmark tools.
kaka11chen opened a new pull request, #21074:
URL: https://github.com/apache/doris/pull/21074
## Proposed changes
Issue Number: close #xxx
Add hdfs benchmark tools.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21074: [Feature](multi-catalog) Add hdfs benchmark tools.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21074:
URL: https://github.com/apache/doris/pull/21074#issuecomment-1600404324
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5331763134") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In bin/run-fs-benchmark.sh line 29:
params="$@"
^--^ SC2124 (warning): Assigning an array to a string! Assign as array, or use * instead of @ to concatenate.
In bin/run-fs-benchmark.sh line 77:
echo "CONF: $CONF"
^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
echo "CONF: ${CONF}"
In bin/run-fs-benchmark.sh line 78:
echo "FS_TYPE: $FS_TYPE"
^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
echo "FS_TYPE: ${FS_TYPE}"
In bin/run-fs-benchmark.sh line 79:
echo "OPERATION: $OPERATION"
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
echo "OPERATION: ${OPERATION}"
In bin/run-fs-benchmark.sh line 80:
echo "THREADS: $THREADS"
^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
echo "THREADS: ${THREADS}"
In bin/run-fs-benchmark.sh line 81:
echo "ITERATIONS: $ITERATIONS"
^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
echo "ITERATIONS: ${ITERATIONS}"
In bin/run-fs-benchmark.sh line 82:
echo "FILE_SIZE: $FILE_SIZE"
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
echo "FILE_SIZE: ${FILE_SIZE}"
In bin/run-fs-benchmark.sh line 84:
echo $params
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
echo "${params}"
In bin/run-fs-benchmark.sh line 231:
if [[ "${RUN_IN_AWS}" -eq 0 ]]; then
^-----------^ SC2154 (warning): RUN_IN_AWS is referenced but not assigned.
In bin/run-fs-benchmark.sh line 329:
${LIMIT:+${LIMIT}} ${DORIS_HOME}/lib/fs_benchmark_tool --conf $CONF --fs_type=$FS_TYPE --operation=$OPERATION --threads=$THREADS --iterations=$ITERATIONS --file_size=$FILE_SIZE 2>&1 | tee "${LOG_DIR}/fs_benchmark_tool.log"
^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^---^ SC2086 (info): Double quote to prevent globbing and word splitting.
^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^---------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
${LIMIT:+${LIMIT}} "${DORIS_HOME}"/lib/fs_benchmark_tool --conf "${CONF}" --fs_type="${FS_TYPE}" --operation="${OPERATION}" --threads="${THREADS}" --iterations="${ITERATIONS}" --file_size="${FILE_SIZE}" 2>&1 | tee "${LOG_DIR}/fs_benchmark_tool.log"
In bin/run-fs-benchmark.sh line 330:
eval `cat ${LOG_DIR}/fs_benchmark_tool.log | grep _median | awk '{printf("qps=%sMB/s, latency=%sms", "'$FILE_SIZE'" / 1024 / 1024 / ($2 * "'$THREADS'" / 1000), $2 * "'$THREADS'")}'`
^-- SC2046 (warning): Quote this to prevent word splitting.
^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
eval $(cat "${LOG_DIR}"/fs_benchmark_tool.log | grep _median | awk '{printf("qps=%sMB/s, latency=%sms", "'"${FILE_SIZE}"'" / 1024 / 1024 / ($2 * "'"${THREADS}"'" / 1000), $2 * "'"${THREADS}"'")}')
In bin/run-fs-benchmark.sh line 335:
echo "thread_num: $THREADS."
^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
echo "thread_num: ${THREADS}."
In bin/run-fs-benchmark.sh line 336:
echo "qps: $qps."
^--^ SC2154 (warning): qps is referenced but not assigned.
^--^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
echo "qps: ${qps}."
In bin/run-fs-benchmark.sh line 337:
echo "latency: $latency."
^------^ SC2154 (warning): latency is referenced but not assigned.
^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean:
echo "latency: ${latency}."
For more information:
https://www.shellcheck.net/wiki/SC2046 -- Quote this to prevent word splitt...
https://www.shellcheck.net/wiki/SC2124 -- Assigning an array to a string! A...
https://www.shellcheck.net/wiki/SC2154 -- RUN_IN_AWS is referenced but not ...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' returned error 1 finding the following formatting issues:
----------
--- bin/run-fs-benchmark.sh.orig
+++ bin/run-fs-benchmark.sh
@@ -28,7 +28,6 @@
params="$@"
-
OPTS="$(getopt \
-n "$0" \
-o '' \
@@ -327,7 +326,7 @@
export JEMALLOC_CONF="percpu_arena:percpu,background_thread:true,metadata_thp:auto,muzzy_decay_ms:30000,dirty_decay_ms:30000,oversize_threshold:0,lg_tcache_max:16,prof_prefix:jeprof.out"
${LIMIT:+${LIMIT}} ${DORIS_HOME}/lib/fs_benchmark_tool --conf $CONF --fs_type=$FS_TYPE --operation=$OPERATION --threads=$THREADS --iterations=$ITERATIONS --file_size=$FILE_SIZE 2>&1 | tee "${LOG_DIR}/fs_benchmark_tool.log"
-eval `cat ${LOG_DIR}/fs_benchmark_tool.log | grep _median | awk '{printf("qps=%sMB/s, latency=%sms", "'$FILE_SIZE'" / 1024 / 1024 / ($2 * "'$THREADS'" / 1000), $2 * "'$THREADS'")}'`
+eval $(cat ${LOG_DIR}/fs_benchmark_tool.log | grep _median | awk '{printf("qps=%sMB/s, latency=%sms", "'$FILE_SIZE'" / 1024 / 1024 / ($2 * "'$THREADS'" / 1000), $2 * "'$THREADS'")}')
echo "------------------------------"
echo " Benchmark Result "
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21074: [Feature](multi-catalog) Add hdfs benchmark tools.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21074:
URL: https://github.com/apache/doris/pull/21074#issuecomment-1605605796
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21074: [Feature](multi-catalog) Add hdfs benchmark tools.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21074:
URL: https://github.com/apache/doris/pull/21074#issuecomment-1605466185
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] kaka11chen commented on pull request #21074: [Feature](multi-catalog) Add hdfs benchmark tools.
Posted by "kaka11chen (via GitHub)" <gi...@apache.org>.
kaka11chen commented on PR #21074:
URL: https://github.com/apache/doris/pull/21074#issuecomment-1605499372
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman merged pull request #21074: [Feature](multi-catalog) Add hdfs benchmark tools.
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #21074:
URL: https://github.com/apache/doris/pull/21074
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21074: [Feature](multi-catalog) Add hdfs benchmark tools.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21074:
URL: https://github.com/apache/doris/pull/21074#issuecomment-1600411248
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21074: [Feature](multi-catalog) Add hdfs benchmark tools.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21074:
URL: https://github.com/apache/doris/pull/21074#issuecomment-1605500104
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21074: [Feature](multi-catalog) Add hdfs benchmark tools.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21074:
URL: https://github.com/apache/doris/pull/21074#issuecomment-1605464211
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/5364423019") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In bin/run-fs-benchmark.sh line 325:
eval $(cat "${LOG_DIR}/fs_benchmark_tool.log" | grep _median | awk '{printf("qps=%sMB/s latency=%sms", "'"${FILE_SIZE}"'" / 1024 / 1024 / ($2 * "'"${THREADS}"'" / 1000), $2 * "'"${THREADS}"'")}')
^-- SC2046 (warning): Quote this to prevent word splitting.
^-- SC2002 (style): Useless cat. Consider 'cmd < file | ..' or 'cmd file | ..' instead.
For more information:
https://www.shellcheck.net/wiki/SC2046 -- Quote this to prevent word splitt...
https://www.shellcheck.net/wiki/SC2002 -- Useless cat. Consider 'cmd < file...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' returned error 1 finding the following formatting issues:
----------
--- bin/run-fs-benchmark.sh.orig
+++ bin/run-fs-benchmark.sh
@@ -26,7 +26,6 @@
export PATH
fi
-
OPTS="$(getopt \
-n "$0" \
-o '' \
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21074: [Feature](multi-catalog) Add hdfs benchmark tools.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21074:
URL: https://github.com/apache/doris/pull/21074#issuecomment-1605605791
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org