You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "nishat-06 (via GitHub)" <gi...@apache.org> on 2026/08/26 07:41:22 UTC
[PR] quote untrusted values in HadoopArchiveLogs.generateScript [hadoop]
nishat-06 opened a new pull request, #8706:
URL: https://github.com/apache/hadoop/pull/8706
### Description of PR
`HadoopArchiveLogs.generateScript` writes the per-application `user` and `appId` values into the generated bash script inside double quotes with no escaping. `user` is the per-user log directory name taken from the remote root log dir listing (`checkFilesAndSeedApps`), and that directory is world-writable on a shared cluster, so a directory name containing a double quote plus shell metacharacters breaks out of the `user="..."` assignment. The script is then run by the archive-logs tool under the yarn/mapred service account, so the crafted name executes as that account. This quotes each interpolated value (`appId`, `user`, `workingDir`, `remoteRootLogDir`, `suffix`) with `Shell.bashQuote`, the same helper already used for shell interpolation in `Shell` and `ShellBasedIdMapping`. `appId` is additionally constrained by the later `ApplicationId.fromString` parse, but `user` is not validated anywhere, so it is the reachable vector.
No JIRA is filed yet since I don't have a JIRA account; happy to update the title and commit to lead with the issue key once one is created.
### How was this patch tested?
Added `testGenerateScriptQuotesUntrustedValues`, which feeds a user directory name carrying an embedded quote and `;touch ...` and asserts the value is emitted as a single bash-quoted token. It fails on the current code and passes after the change. Updated `testGenerateScript` for the quoted output, ran the `TestHadoopArchiveLogs` suite on JDK 17 (7/7 green), and checkstyle on the touched files is clean.
### For code changes:
- [ ] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the endpoint
declared according to the connector-specific documentation? *Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.*
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ ] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] quote untrusted values in HadoopArchiveLogs.generateScript [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8706:
URL: https://github.com/apache/hadoop/pull/8706#issuecomment-5422891596
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 27m 18s | | trunk passed |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | spotbugs | 0m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 47s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 17m 5s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 13s | | the patch passed |
| +1 :green_heart: | compile | 0m 14s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 0m 13s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 0m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 11s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 13s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 13s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | spotbugs | 0m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 17s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 41s | | hadoop-archive-logs in the patch passed. |
| -1 :x: | asflicense | 0m 25s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8706/1/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 68m 46s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8706/1/artifact/out/Dockerfile |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c7cf47405dda 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 33237971fa49f7ba0a0f4e2790dfdd7b792e8acc |
| Default Java | Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8706/1/testReport/ |
| Max. process+thread count | 634 (vs. ulimit of 10000) |
| modules | C: hadoop-tools/hadoop-archive-logs U: hadoop-tools/hadoop-archive-logs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8706/1/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Tune HadoopArchiveLogs [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8706:
URL: https://github.com/apache/hadoop/pull/8706#issuecomment-5471607150
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 28m 37s | | trunk passed |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | compile | 0m 23s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | spotbugs | 0m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 57s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 17m 15s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 12s | | the patch passed |
| +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 0m 12s | | the patch passed |
| +1 :green_heart: | compile | 0m 14s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 0m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 8s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 14s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 13s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | spotbugs | 0m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 46s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 44s | | hadoop-archive-logs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not generate ASF License warnings. |
| | | 69m 47s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8706/3/artifact/out/Dockerfile |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b02e45ea16d0 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / abbc78d44168ead0a604b49b195fe677931a8cd2 |
| Default Java | Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8706/3/testReport/ |
| Max. process+thread count | 612 (vs. ulimit of 10000) |
| modules | C: hadoop-tools/hadoop-archive-logs U: hadoop-tools/hadoop-archive-logs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8706/3/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Tune HadoopArchiveLogs [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8706:
URL: https://github.com/apache/hadoop/pull/8706#issuecomment-5471595099
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 28m 34s | | trunk passed |
| +1 :green_heart: | compile | 0m 26s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | compile | 0m 26s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | spotbugs | 0m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 15s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 16m 34s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 15s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 0m 15s | | the patch passed |
| +1 :green_heart: | compile | 0m 13s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 0m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 8s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 13s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 11s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | spotbugs | 0m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 38s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 42s | | hadoop-archive-logs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 23s | | The patch does not generate ASF License warnings. |
| | | 69m 48s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8706/2/artifact/out/Dockerfile |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 214e16ef8a4e 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / abbc78d44168ead0a604b49b195fe677931a8cd2 |
| Default Java | Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8706/2/testReport/ |
| Max. process+thread count | 636 (vs. ulimit of 10000) |
| modules | C: hadoop-tools/hadoop-archive-logs U: hadoop-tools/hadoop-archive-logs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8706/2/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Tune HadoopArchiveLogs [hadoop]
Posted by "Hexiaoqiao (via GitHub)" <gi...@apache.org>.
Hexiaoqiao commented on PR #8706:
URL: https://github.com/apache/hadoop/pull/8706#issuecomment-5578774867
@nishat-06 Thanks for your contribution, please file a JIAR first here https://issues.apache.org/jira/projects/HDFS/issues and then connect this PR and JIRA. Thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] quote untrusted values in HadoopArchiveLogs.generateScript [hadoop]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #8706:
URL: https://github.com/apache/hadoop/pull/8706#issuecomment-5467312848
@steveloughran Hi Steve, could I ask for your advice on how we should handle [PR #8706]
The PR describes a potential command-execution issue across a privilege boundary.
According to Hadoop’s `SECURITY.md`, an unfixed vulnerability should normally be reported privately rather than through a public PR or JIRA.
Should we ask the author to contact `security@hadoop.apache.org` and continue the technical discussion privately? Should the current PR be closed or otherwise handled differently until the security team completes its triage? Alternatively, has this issue already been assessed as a normal bug that can continue through the public workflow?
Thanks for your guidance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org