[I] Source-release binary-file check never runs: 'file -I' is a BSD-only flag that GNU file rejects [hudi]

["wombatu-kun (via GitHub)" <gi...@apache.org>]

wombatu-kun opened a new issue, #19266:
URL: https://github.com/apache/hudi/issues/19266

   `scripts/release/validate_source_binary_files.sh` is meant to keep binary files out of the Apache source release. It has never fired on Linux, which is every CI runner and most release managers' machines.
   
   ### The flag
   
   The check pipes every file through `file -I`:
   
       numBinaryFiles=`find . -iname '*' | xargs -I {} file -I {} | grep -va directory | ... | wc -l`
   
   `-I` is the BSD/macOS spelling of the MIME option. GNU `file` rejects it:
   
       $ file -I pom.xml
       file: invalid option -- 'I'
       Usage: file [-bcCdEhikLlNnprsSvzZ0] [--apple] [--extension] [--mime-encoding] ...
       $ echo $?
       1
   
   It prints usage to stderr and nothing to stdout, so `wc -l` counts zero and `numBinaryFiles` is always `0`. `set -o errexit` does not catch it: the pipeline sits inside a command substitution in an assignment, and without `pipefail` the assignment takes the exit status of the last stage (`sed`), which is 0.
   
   Confirmed by running the current script, unmodified, against a clean checkout of master, whose tree contains 105 image files outside `src/test/`: it exits 0 and reports `No Binary Files in the source files? - [OK]`.
   
   ### The directory
   
   Fixing only the flag turns CI red, because the check is also invoked in the wrong place. Its contract is "run me from the root of a source release tree" - that is how `validate_staged_release.sh` calls it, after `cd`-ing into the extracted tarball. But the `validate-source` job in `bot.yml` runs it against the raw git checkout, which still contains `rfc/`, `docker/images/`, `hudi-notebooks/notebooks/common/images/` and `.idea/` - all of them deliberately excluded from the source release by `create_source_directory.sh`.
   
   A working check reports:
   
   | Where it runs | Files flagged on current master |
   | --- | --- |
   | Raw git checkout (what `bot.yml` does today) | 105 |
   | Source release directory (what the script is written for) | 0 |
   
   ### Consequence
   
   Nothing has stopped a binary from entering the source release. #19265 adds `hudi-agent-gateway/src/hudi_agent_gateway/ui/hudi-logo.png`, which `create_source_directory.sh` does not exclude, and no check went red. Found while reviewing that PR: https://github.com/apache/hudi/pull/19265#discussion_r3567769689
   
   ### Fix
   
   Ask libmagic for the charset rather than the MIME type (`file --mime`, the long option that both GNU and BSD accept; `charset=binary` is a direct answer that does not depend on libmagic's growing type vocabulary), run the check inside the source release directory in `bot.yml`, and add `set -o pipefail` so a broken `file` invocation can never again read as "no binary files found".
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Source-release binary-file check never runs: 'file -I' is a BSD-only flag that GNU file rejects [hudi]

["voonhous (via GitHub)" <gi...@apache.org>]

voonhous closed issue #19266: Source-release binary-file check never runs: 'file -I' is a BSD-only flag that GNU file rejects
URL: https://github.com/apache/hudi/issues/19266


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org