You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "hiroyuki-sato (via GitHub)" <gi...@apache.org> on 2026/08/31 01:48:19 UTC
[PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
hiroyuki-sato opened a new pull request, #51091:
URL: https://github.com/apache/arrow/pull/51091
### Rationale for this change
This is the sub issue #44748.
* SC2046: Quote this to prevent word splitting.
* SC2128: Expanding an array without an index only gives the element in the index 0.
* SC2086: Double quote to prevent globbing and word splitting.
```
In generate_corpuses.sh line 32:
ARROW_ROOT=$(cd $(dirname "$BASH_SOURCE")/../../..; pwd)
^-----------------------^ SC2046 (warning): Quote this to prevent word splitting.
^----------^ SC2128 (warning): Expanding an array without an index only gives the first element.
In generate_corpuses.sh line 43:
${OUT}/arrow-ipc-generate-fuzz-corpus -stream ${CORPUS_DIR}
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"${OUT}"/arrow-ipc-generate-fuzz-corpus -stream ${CORPUS_DIR}
In generate_corpuses.sh line 45:
IPC_INTEGRATION_FILES=$(find ${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration -name "*.stream")
^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
IPC_INTEGRATION_FILES=$(find "${ARROW_ROOT}"/testing/data/arrow-ipc-stream/integration -name "*.stream")
In generate_corpuses.sh line 49:
cp --backup=numbered ${IPC_INTEGRATION_FILES} ${CORPUS_DIR}
^----------------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
cp --backup=numbered "${IPC_INTEGRATION_FILES}" ${CORPUS_DIR}
In generate_corpuses.sh line 50:
${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/arrow-ipc-stream-fuzz_seed_corpus.zip
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"${ARROW_CPP}"/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} "${OUT}"/arrow-ipc-stream-fuzz_seed_corpus.zip
In generate_corpuses.sh line 53:
${OUT}/arrow-ipc-generate-fuzz-corpus -file ${CORPUS_DIR}
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"${OUT}"/arrow-ipc-generate-fuzz-corpus -file ${CORPUS_DIR}
In generate_corpuses.sh line 54:
IPC_INTEGRATION_FILES=$(find ${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration -name "*.arrow_file")
^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
IPC_INTEGRATION_FILES=$(find "${ARROW_ROOT}"/testing/data/arrow-ipc-stream/integration -name "*.arrow_file")
In generate_corpuses.sh line 56:
cp --backup=numbered ${IPC_INTEGRATION_FILES} ${CORPUS_DIR}
^----------------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
cp --backup=numbered "${IPC_INTEGRATION_FILES}" ${CORPUS_DIR}
In generate_corpuses.sh line 57:
${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/arrow-ipc-file-fuzz_seed_corpus.zip
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"${ARROW_CPP}"/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} "${OUT}"/arrow-ipc-file-fuzz_seed_corpus.zip
In generate_corpuses.sh line 60:
${OUT}/arrow-ipc-generate-tensor-fuzz-corpus -stream ${CORPUS_DIR}
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"${OUT}"/arrow-ipc-generate-tensor-fuzz-corpus -stream ${CORPUS_DIR}
In generate_corpuses.sh line 61:
${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/arrow-ipc-tensor-stream-fuzz_seed_corpus.zip
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"${ARROW_CPP}"/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} "${OUT}"/arrow-ipc-tensor-stream-fuzz_seed_corpus.zip
In generate_corpuses.sh line 66:
${OUT}/parquet-arrow-generate-fuzz-corpus ${CORPUS_DIR}
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"${OUT}"/parquet-arrow-generate-fuzz-corpus ${CORPUS_DIR}
In generate_corpuses.sh line 68:
cp ${ARROW_CPP}/submodules/parquet-testing/data/*.parquet ${CORPUS_DIR}
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
cp "${ARROW_CPP}"/submodules/parquet-testing/data/*.parquet ${CORPUS_DIR}
In generate_corpuses.sh line 69:
cp ${ARROW_CPP}/submodules/parquet-testing/bad_data/*.parquet ${CORPUS_DIR}
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
cp "${ARROW_CPP}"/submodules/parquet-testing/bad_data/*.parquet ${CORPUS_DIR}
In generate_corpuses.sh line 70:
${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/parquet-arrow-fuzz_seed_corpus.zip
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"${ARROW_CPP}"/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} "${OUT}"/parquet-arrow-fuzz_seed_corpus.zip
In generate_corpuses.sh line 75:
${OUT}/parquet-generate-encoding-fuzz-corpus ${CORPUS_DIR}
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"${OUT}"/parquet-generate-encoding-fuzz-corpus ${CORPUS_DIR}
In generate_corpuses.sh line 76:
${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/parquet-encoding-fuzz_seed_corpus.zip
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"${ARROW_CPP}"/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} "${OUT}"/parquet-encoding-fuzz_seed_corpus.zip
In generate_corpuses.sh line 84:
${OUT}/arrow-csv-generate-fuzz-corpus ${CORPUS_DIR}
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"${OUT}"/arrow-csv-generate-fuzz-corpus ${CORPUS_DIR}
In generate_corpuses.sh line 86:
cp ${ARROW_ROOT}/testing/data/csv/*.csv ${CORPUS_DIR}
^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
cp "${ARROW_ROOT}"/testing/data/csv/*.csv ${CORPUS_DIR}
In generate_corpuses.sh line 89:
${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/arrow-csv-fuzz_seed_corpus.zip
^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
"${ARROW_CPP}"/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} "${OUT}"/arrow-csv-fuzz_seed_corpus.zip
For more information:
https://www.shellcheck.net/wiki/SC2046 -- Quote this to prevent word splitt...
https://www.shellcheck.net/wiki/SC2128 -- Expanding an array without an ind...
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
```
### What changes are included in this PR?
* SC2046: Quote command substitutions to prevent word splitting.
* SC2128: Specify an array index explicitly.
* SC2086: Quote variables to prevent globbing and word splitting.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #51091:
URL: https://github.com/apache/arrow/pull/51091#issuecomment-5488287834
Yes if you verify it on local.
AMD64 Ubuntu 24.04 C++ ASAN UBSAN also uses it:
https://github.com/apache/arrow/actions/runs/33349187376/job/99359104165?pr=51091#step:8:4543
```text
+ /arrow/cpp/build-support/fuzzing/generate_corpuses.sh /build/cpp/debug
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou merged PR #51091:
URL: https://github.com/apache/arrow/pull/51091
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #51091:
URL: https://github.com/apache/arrow/pull/51091#discussion_r3946439724
##########
cpp/build-support/fuzzing/generate_corpuses.sh:
##########
@@ -39,51 +39,59 @@ OUT=$1
# Arrow IPC
-rm -rf ${CORPUS_DIR}
-${OUT}/arrow-ipc-generate-fuzz-corpus -stream ${CORPUS_DIR}
+rm -rf "${CORPUS_DIR}"
+"${OUT}/arrow-ipc-generate-fuzz-corpus" -stream "${CORPUS_DIR}"
+
+IPC_INTEGRATION_DIR="${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration"
+
# Add "golden" IPC integration files
-IPC_INTEGRATION_FILES=$(find ${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration -name "*.stream")
-[ -z "${IPC_INTEGRATION_FILES}" ] && exit 1
# Several IPC integration files can have the same name, make sure
# they all appear in the corpus by numbering the duplicates.
-cp --backup=numbered ${IPC_INTEGRATION_FILES} ${CORPUS_DIR}
-${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/arrow-ipc-stream-fuzz_seed_corpus.zip
+# Use shell globbing to keep this ShellCheck-safe and compatible with Bash 3.2 on macOS.
+# IPC integration files are expected to be directly under each subdirectory.
+cp --backup=numbered \
+ "${IPC_INTEGRATION_DIR}"/*/*.stream \
+ "${CORPUS_DIR}"
Review Comment:
Can we use `find ${IPC_INTEGRATION_DIR} -name "*.stream" -exec cp --backup=numbered '{}' "${CORPUS_DIR}" \;` instead of `while` and `read` because we removed `[ "${#IPC_INTEGRATION_FILES[@]}" -eq 0 ] && exit 1`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "hiroyuki-sato (via GitHub)" <gi...@apache.org>.
hiroyuki-sato commented on PR #51091:
URL: https://github.com/apache/arrow/pull/51091#issuecomment-5494046185
I forgot that macOS uses Bash 3.2, which doesn't support the `mapfile` command. I'll rewirte later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "hiroyuki-sato (via GitHub)" <gi...@apache.org>.
hiroyuki-sato commented on PR #51091:
URL: https://github.com/apache/arrow/pull/51091#issuecomment-5473022690
@github-actions crossbow submit test-build-cpp-fuzz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "hiroyuki-sato (via GitHub)" <gi...@apache.org>.
hiroyuki-sato commented on PR #51091:
URL: https://github.com/apache/arrow/pull/51091#issuecomment-5488382156
@kou Thanks. Oh, I didn't know that `AMD64 Ubuntu 24.04 C++ ASAN UBSAN` runs with `ARROW_FUZZING=ON`.
I'll take a look at the failure logs.
I'll mark this PR as a draft for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "hiroyuki-sato (via GitHub)" <gi...@apache.org>.
hiroyuki-sato commented on PR #51091:
URL: https://github.com/apache/arrow/pull/51091#issuecomment-5559757958
@kou `mapfile` is not available in Bash 3.2 on macOS, so I simplified the implementation. Please take a look.
The CI failures are unrelated to this PR, and some of them may be fixed by #51012.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51091:
URL: https://github.com/apache/arrow/pull/51091#issuecomment-5473035331
Revision: 534ce6f557c20da41a97ea855db8ba5d40f8562e
Submitted crossbow builds: [ursacomputing/crossbow @ actions-08167fab61](https://github.com/ursacomputing/crossbow/branches/all?query=actions-08167fab61)
|Task|Status|
|----|------|
|test-build-cpp-fuzz|[](https://github.com/ursacomputing/crossbow/actions/runs/33351433497/job/99365465807)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "hiroyuki-sato (via GitHub)" <gi...@apache.org>.
hiroyuki-sato commented on PR #51091:
URL: https://github.com/apache/arrow/pull/51091#issuecomment-5491178784
@kou Could you take a look when you get a chacne? I think the two CI failures are unrelated to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #51091:
URL: https://github.com/apache/arrow/pull/51091#discussion_r3945904926
##########
cpp/build-support/fuzzing/generate_corpuses.sh:
##########
@@ -39,51 +39,59 @@ OUT=$1
# Arrow IPC
-rm -rf ${CORPUS_DIR}
-${OUT}/arrow-ipc-generate-fuzz-corpus -stream ${CORPUS_DIR}
+rm -rf "${CORPUS_DIR}"
+"${OUT}/arrow-ipc-generate-fuzz-corpus" -stream "${CORPUS_DIR}"
+
+IPC_INTEGRATION_DIR="${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration"
+
# Add "golden" IPC integration files
-IPC_INTEGRATION_FILES=$(find ${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration -name "*.stream")
-[ -z "${IPC_INTEGRATION_FILES}" ] && exit 1
# Several IPC integration files can have the same name, make sure
# they all appear in the corpus by numbering the duplicates.
-cp --backup=numbered ${IPC_INTEGRATION_FILES} ${CORPUS_DIR}
-${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/arrow-ipc-stream-fuzz_seed_corpus.zip
+# Use shell globbing to keep this ShellCheck-safe and compatible with Bash 3.2 on macOS.
+# IPC integration files are expected to be directly under each subdirectory.
+cp --backup=numbered \
+ "${IPC_INTEGRATION_DIR}"/*/*.stream \
+ "${CORPUS_DIR}"
Review Comment:
I think that we can remove the `[ "${#IPC_INTEGRATION_FILES[@]}" -eq 0 ]` check. If we don't have it, can we simplify this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "hiroyuki-sato (via GitHub)" <gi...@apache.org>.
hiroyuki-sato commented on code in PR #51091:
URL: https://github.com/apache/arrow/pull/51091#discussion_r3946685738
##########
cpp/build-support/fuzzing/generate_corpuses.sh:
##########
@@ -39,51 +39,59 @@ OUT=$1
# Arrow IPC
-rm -rf ${CORPUS_DIR}
-${OUT}/arrow-ipc-generate-fuzz-corpus -stream ${CORPUS_DIR}
+rm -rf "${CORPUS_DIR}"
+"${OUT}/arrow-ipc-generate-fuzz-corpus" -stream "${CORPUS_DIR}"
+
+IPC_INTEGRATION_DIR="${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration"
+
# Add "golden" IPC integration files
-IPC_INTEGRATION_FILES=$(find ${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration -name "*.stream")
-[ -z "${IPC_INTEGRATION_FILES}" ] && exit 1
# Several IPC integration files can have the same name, make sure
# they all appear in the corpus by numbering the duplicates.
-cp --backup=numbered ${IPC_INTEGRATION_FILES} ${CORPUS_DIR}
-${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/arrow-ipc-stream-fuzz_seed_corpus.zip
+# Use shell globbing to keep this ShellCheck-safe and compatible with Bash 3.2 on macOS.
+# IPC integration files are expected to be directly under each subdirectory.
+cp --backup=numbered \
+ "${IPC_INTEGRATION_DIR}"/*/*.stream \
+ "${CORPUS_DIR}"
Review Comment:
Sorry, I misunderstood your comment. I change to use `find..-exec`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #51091:
URL: https://github.com/apache/arrow/pull/51091#issuecomment-5488037697
It seems that we have a document for it: https://arrow.apache.org/docs/developers/cpp/fuzzing.html
`archery docker run ubuntu-cpp-sanitizer` will work too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51091:
URL: https://github.com/apache/arrow/pull/51091#issuecomment-5472705585
:warning: GitHub issue #51090 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #51091:
URL: https://github.com/apache/arrow/pull/51091#discussion_r3945672822
##########
cpp/build-support/fuzzing/generate_corpuses.sh:
##########
@@ -39,51 +39,59 @@ OUT=$1
# Arrow IPC
-rm -rf ${CORPUS_DIR}
-${OUT}/arrow-ipc-generate-fuzz-corpus -stream ${CORPUS_DIR}
+rm -rf "${CORPUS_DIR}"
+"${OUT}/arrow-ipc-generate-fuzz-corpus" -stream "${CORPUS_DIR}"
+
+IPC_INTEGRATION_DIR="${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration"
+
# Add "golden" IPC integration files
-IPC_INTEGRATION_FILES=$(find ${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration -name "*.stream")
-[ -z "${IPC_INTEGRATION_FILES}" ] && exit 1
# Several IPC integration files can have the same name, make sure
# they all appear in the corpus by numbering the duplicates.
-cp --backup=numbered ${IPC_INTEGRATION_FILES} ${CORPUS_DIR}
-${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/arrow-ipc-stream-fuzz_seed_corpus.zip
+# Use shell globbing to keep this ShellCheck-safe and compatible with Bash 3.2 on macOS.
+# IPC integration files are expected to be directly under each subdirectory.
+cp --backup=numbered \
+ "${IPC_INTEGRATION_DIR}"/*/*.stream \
+ "${CORPUS_DIR}"
Review Comment:
Can we use `find ${IPC_INTEGRATION_DIR} -name "*.stream" -exec cp --backup=numbered '{}' "${CORPUS_DIR}" \;` or something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "hiroyuki-sato (via GitHub)" <gi...@apache.org>.
hiroyuki-sato commented on code in PR #51091:
URL: https://github.com/apache/arrow/pull/51091#discussion_r3945880844
##########
cpp/build-support/fuzzing/generate_corpuses.sh:
##########
@@ -39,51 +39,59 @@ OUT=$1
# Arrow IPC
-rm -rf ${CORPUS_DIR}
-${OUT}/arrow-ipc-generate-fuzz-corpus -stream ${CORPUS_DIR}
+rm -rf "${CORPUS_DIR}"
+"${OUT}/arrow-ipc-generate-fuzz-corpus" -stream "${CORPUS_DIR}"
+
+IPC_INTEGRATION_DIR="${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration"
+
# Add "golden" IPC integration files
-IPC_INTEGRATION_FILES=$(find ${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration -name "*.stream")
-[ -z "${IPC_INTEGRATION_FILES}" ] && exit 1
# Several IPC integration files can have the same name, make sure
# they all appear in the corpus by numbering the duplicates.
-cp --backup=numbered ${IPC_INTEGRATION_FILES} ${CORPUS_DIR}
-${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/arrow-ipc-stream-fuzz_seed_corpus.zip
+# Use shell globbing to keep this ShellCheck-safe and compatible with Bash 3.2 on macOS.
+# IPC integration files are expected to be directly under each subdirectory.
+cp --backup=numbered \
+ "${IPC_INTEGRATION_DIR}"/*/*.stream \
+ "${CORPUS_DIR}"
Review Comment:
Thank you for your review. Change to use `while` .. `read` instead of glob like the following
```bash
# Add "golden" IPC integration files
# Use read instead of mapfile to keep this compatible with Bash 3.2 on macOS.
IPC_INTEGRATION_FILES=()
while IFS= read -r -d '' IPC_INTEGRATION_FILE; do
IPC_INTEGRATION_FILES+=("${IPC_INTEGRATION_FILE}")
done < <(
find "${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration" \
-name "*.stream" -print0
)
# Exit with an error if find returns no files.
[ "${#IPC_INTEGRATION_FILES[@]}" -eq 0 ] && exit 1
# Several IPC integration files can have the same name, make sure
# they all appear in the corpus by numbering the duplicates.
cp --backup=numbered "${IPC_INTEGRATION_FILES[@]}" "${CORPUS_DIR}"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "hiroyuki-sato (via GitHub)" <gi...@apache.org>.
hiroyuki-sato commented on code in PR #51091:
URL: https://github.com/apache/arrow/pull/51091#discussion_r3946086289
##########
cpp/build-support/fuzzing/generate_corpuses.sh:
##########
@@ -39,51 +39,59 @@ OUT=$1
# Arrow IPC
-rm -rf ${CORPUS_DIR}
-${OUT}/arrow-ipc-generate-fuzz-corpus -stream ${CORPUS_DIR}
+rm -rf "${CORPUS_DIR}"
+"${OUT}/arrow-ipc-generate-fuzz-corpus" -stream "${CORPUS_DIR}"
+
+IPC_INTEGRATION_DIR="${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration"
+
# Add "golden" IPC integration files
-IPC_INTEGRATION_FILES=$(find ${ARROW_ROOT}/testing/data/arrow-ipc-stream/integration -name "*.stream")
-[ -z "${IPC_INTEGRATION_FILES}" ] && exit 1
# Several IPC integration files can have the same name, make sure
# they all appear in the corpus by numbering the duplicates.
-cp --backup=numbered ${IPC_INTEGRATION_FILES} ${CORPUS_DIR}
-${ARROW_CPP}/build-support/fuzzing/pack_corpus.py ${CORPUS_DIR} ${OUT}/arrow-ipc-stream-fuzz_seed_corpus.zip
+# Use shell globbing to keep this ShellCheck-safe and compatible with Bash 3.2 on macOS.
+# IPC integration files are expected to be directly under each subdirectory.
+cp --backup=numbered \
+ "${IPC_INTEGRATION_DIR}"/*/*.stream \
+ "${CORPUS_DIR}"
Review Comment:
Thank you for your comment. Removed the following parts.
```bash
# Exit with an error if find returns no files.
[ "${#IPC_INTEGRATION_FILES[@]}" -eq 0 ] && exit 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "hiroyuki-sato (via GitHub)" <gi...@apache.org>.
hiroyuki-sato commented on PR #51091:
URL: https://github.com/apache/arrow/pull/51091#issuecomment-5487657967
@kou
It looks like `generate_corpuses.sh` is called when `"${ARROW_FUZZING}" == "ON"`, but I don't know how to enable `ARROW_FUZZING`.
```bash
if [ "${ARROW_FUZZING}" == "ON" ]; then
# .. snip
# 1. Generate seed corpuses
# For IPC fuzz targets, these will include the golden IPC integration files.
"${source_dir}/build-support/fuzzing/generate_corpuses.sh" "${binary_output_dir}"
```
```
`ARROW_FUZZING` is enabled in the following parts.
cpp/CMakePresets.json
```json
{
"name": "fuzzing",
"inherits": [
"sanitizer-asan",
"sanitizer-ubsan",
"base"
],
"displayName": "Debug build with IPC and Parquet fuzzing targets",
"cacheVariables": {
"ARROW_CSV": "ON",
"ARROW_DEPENDENCY_SOURCE": "BUNDLED",
"ARROW_FUZZING": "ON",
```
ubuntu-cpp-sanitizer
```
ubuntu-cpp-sanitizer:
#..snip
environment:
<<: [*common, *ccache, *sccache, *cpp]
# snip...
ARROW_FLIGHT_SQL: "OFF"
ARROW_FUZZING: "ON" # Check fuzz regressions
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51090: [CI][C++] Fix shellcheck errors in cpp/build-support/fuzzing/generate_corpuses.sh [arrow]
Posted by "hiroyuki-sato (via GitHub)" <gi...@apache.org>.
hiroyuki-sato commented on PR #51091:
URL: https://github.com/apache/arrow/pull/51091#issuecomment-5488149481
@kou Thanks. Does this mean I need to run Archery on my local machine instead of in CI?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org