You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "K0K0V0K (via GitHub)" <gi...@apache.org> on 2025/11/22 15:50:22 UTC
[PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
K0K0V0K opened a new pull request, #8100:
URL: https://github.com/apache/hadoop/pull/8100
### Description of PR
The goal of this feature tp provide a configurable mechanism to control which users are allowed to execute specific MapReduce jobs. This feature aims to prevent unauthorized or potentially harmful mapper/reducer implementations from running within the Hadoop cluster.
In the standard Hadoop MapReduce execution flow:
1) A MapReduce job is submitted by a user.
2) The job is registered with the Resource Manager (RM). 3) The RM assigns the job to a Node Manager (NM), where the Application Master (AM) for the job is launched. 4) The AM requests additional containers from the cluster, to be able to start tasks. 5) The NM launches those containers, and the containers execute the mapper/reducer tasks defined by the job.
The proposed feature introduces a security filtering mechanism inside the Application Master. Before mapper or reducer tasks are launched, the AM will verify that the user-submitted MapReduce code complies with a cluster-defined security policy. This ensures that only approved classes or packages can be executed inside the containers. The goal is to protect the cluster from unwanted or unsafe task implementations, such as custom code that may introduce performance, stability, or security risks.
Upon receiving job metadata, the Application Master will: 1) Check the feature is enabled.
2) Check the user who submitted the job is allowed to bypass the security check. 3) Compare classes in job config against the denied task list. 4) If job is not authorised an exception will be thrown and AM will fail.
New Configs
Enables MapReduce Task-Level Security Enforcement
When enabled, the Application Master performs validation of user-submitted mapper, reducer, and other task-related classes before launching containers. This mechanism protects the cluster from running disallowed or unsafe task implementations as defined by administrator-controlled policies.
- Property name: mapreduce.security.enabled
- Property type: boolean
- Default: false (security disabled)
MapReduce Task-Level Security Enforcement: Property Domain Defines the set of MapReduce configuration keys that represent user-supplied class names involved in task execution (e.g., mapper, reducer, partitioner). The Application Master examines the values of these properties and checks whether any referenced class is listed in denied tasks. Administrators may override this list to expand or restrict the validation domain.
- Property name: mapreduce.security.property-domain
- Property type: list of configuration keys
- Default: map.sort.class
mapreduce.job.classloader.system.classes
mapreduce.job.combine.class
mapreduce.job.combiner.group.comparator.class
mapreduce.job.end-notification.custom-notifier-class mapreduce.job.inputformat.class
mapreduce.job.map.class
mapreduce.job.map.output.collector.class
mapreduce.job.output.group.comparator.class
mapreduce.job.output.key.class
mapreduce.job.output.key.comparator.class
mapreduce.job.output.value.class
mapreduce.job.outputformat.class
mapreduce.job.partitioner.class
mapreduce.job.reduce.class
mapreduce.map.output.key.class
mapreduce.map.output.value.class
MapReduce Task-Level Security Enforcement: Denied Tasks Specifies the list of disallowed task implementation classes or packages. If a user submits a job whose mapper, reducer, or other task-related classes match any entry in this blacklist.
- Property name: mapreduce.security.denied-tasks
- Property type: list of class name or package patterns
- Default: empty
- Example: org.apache.hadoop.streaming,org.apache.hadoop.examples.QuasiMonteCarlo
MapReduce Task-Level Security Enforcement: Allowed Users Specifies users who may bypass the blacklist defined in denied tasks. This whitelist is intended for trusted or system-level workflows that may legitimately require the use of restricted task implementations. If the submitting user is listed here, blacklist enforcement is skipped, although standard Hadoop authentication and ACL checks still apply.
- Property name: mapreduce.security.allowed-users
- Property type: list of usernames
- Default: empty
- Example: alice,bob
### How was this patch tested?
UT was run
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-3567011796
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 16m 2s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 8m 9s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 34s | | trunk passed |
| +1 :green_heart: | compile | 15m 43s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 16m 9s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 3m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 24s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 15s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +0 :ok: | spotbugs | 0m 45s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 1m 48s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/2/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 178 extant spotbugs warnings. |
| -1 :x: | spotbugs | 1m 20s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/2/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-warnings.html) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in trunk has 39 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 26m 35s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 15m 11s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 15m 11s | | the patch passed |
| +1 :green_heart: | compile | 16m 15s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 16m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 12s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/2/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 101 unchanged - 0 fixed = 103 total (was 101) |
| +1 :green_heart: | mvnsite | 2m 44s | | the patch passed |
| -1 :x: | javadoc | 0m 51s | [/results-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/2/artifact/out/results-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 1 new + 1124 unchanged - 0 fixed = 1125 total (was 1124) |
| +1 :green_heart: | javadoc | 2m 16s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +0 :ok: | spotbugs | 0m 42s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 27m 0s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 40s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 8m 18s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 8m 18s | | hadoop-mapreduce-client-app in the patch passed. |
| -1 :x: | asflicense | 0m 59s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/2/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 224m 37s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8100 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux 2046eddf5e24 5.15.0-160-generic #170-Ubuntu SMP Wed Oct 1 10:06:56 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8c688bb887180d269effe183bffd9dc576c5b42e |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/2/testReport/ |
| Max. process+thread count | 1590 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/2/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-4119430349
I think we was waiting on the 8372 pr.
I will retrigger the flow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-3567051454
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 15s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 8m 2s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 22s | | trunk passed |
| +1 :green_heart: | compile | 17m 53s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 18m 10s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 3m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 12s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 7s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +0 :ok: | spotbugs | 0m 42s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 1m 52s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/5/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 178 extant spotbugs warnings. |
| -1 :x: | spotbugs | 1m 22s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/5/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-warnings.html) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in trunk has 39 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 31m 48s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 16m 53s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 16m 53s | | the patch passed |
| +1 :green_heart: | compile | 18m 8s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 18m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 35s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/5/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 101 unchanged - 0 fixed = 103 total (was 101) |
| +1 :green_heart: | mvnsite | 2m 36s | | the patch passed |
| -1 :x: | javadoc | 0m 47s | [/results-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/5/artifact/out/results-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 1 new + 1124 unchanged - 0 fixed = 1125 total (was 1124) |
| +1 :green_heart: | javadoc | 2m 6s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +0 :ok: | spotbugs | 0m 38s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 31m 50s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 37s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 8m 52s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 8m 28s | | hadoop-mapreduce-client-app in the patch passed. |
| -1 :x: | asflicense | 1m 3s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/5/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 234m 48s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8100 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux cc3b4123f2d8 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 40457a3359ccdd39ade9412eed077d166c960efa |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/5/testReport/ |
| Max. process+thread count | 1092 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/5/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-4112748607
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 20m 5s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 14s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 57m 23s | | trunk passed |
| +1 :green_heart: | compile | 19m 28s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | compile | 21m 4s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | checkstyle | 7m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 20s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 2m 20s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +0 :ok: | spotbugs | 0m 42s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 0m 54s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/2/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt) | hadoop-mapreduce-client-app in trunk failed. |
| -1 :x: | spotbugs | 1m 2s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/2/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in trunk failed. |
| +1 :green_heart: | shadedclient | 34m 44s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 17m 27s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javac | 17m 27s | | the patch passed |
| +1 :green_heart: | compile | 21m 7s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | javac | 21m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 6m 55s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 17s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 2m 11s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +0 :ok: | spotbugs | 0m 41s | | hadoop-project has no data from spotbugs |
| -1 :x: | spotbugs | 1m 2s | [/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/2/artifact/out/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | spotbugs | 0m 55s | [/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/2/artifact/out/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt) | hadoop-mapreduce-client-app in the patch failed. |
| +1 :green_heart: | shadedclient | 34m 48s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 39s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 9m 4s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 9m 10s | | hadoop-mapreduce-client-app in the patch passed. |
| +1 :green_heart: | asflicense | 1m 12s | | The patch does not generate ASF License warnings. |
| | | 293m 14s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8100 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux efcbe61c0eac 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ee7a5db96b6661370ee0872aa46d0fab4ca16d65 |
| Default Java | Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/2/testReport/ |
| Max. process+thread count | 1078 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/2/console |
| versions | git=2.43.0 maven=3.9.11 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2607437440
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/TaskLevelSecurityEnforcement.md:
##########
@@ -0,0 +1,92 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+MR Task-Level Security Enforcement
+==================
+
+<!-- MACRO{toc|fromDepth=0|toDepth=2} -->
+
+Overview
+-------
+The goal of this feature tp provide a configurable mechanism to control which users are allowed to execute specific MapReduce jobs.
+This feature aims to prevent unauthorized or potentially harmful mapper/reducer implementations from running within the Hadoop cluster.
+
+In the standard Hadoop MapReduce execution flow:
+1) A MapReduce job is submitted by a user.
+2) The job is registered with the Resource Manager (RM).
+3) The RM assigns the job to a Node Manager (NM), where the Application Master (AM) for the job is launched.
+4) The AM requests additional containers from the cluster, to be able to start tasks.
+5) The NM launches those containers, and the containers execute the mapper/reducer tasks defined by the job.
+
+This feature introduces a security filtering mechanism inside the Application Master.
+Before mapper or reducer tasks are launched, the AM will verify that the user-submitted MapReduce code complies with a cluster-defined security policy.
+This ensures that only approved classes or packages can be executed inside the containers.
+The goal is to protect the cluster from unwanted or unsafe task implementations, such as custom code that may introduce performance, stability, or security risks.
+
+Upon receiving job metadata, the Application Master will:
+1) Check the feature is enabled.
+2) Check the user who submitted the job is allowed to bypass the security check.
+3) Compare classes in job config against the denied task list.
+4) If job is not authorised an exception will be thrown and AM will fail.
+
+Configurations
+-------
+
+#### Enables MapReduce Task-Level Security Enforcement
+When enabled, the Application Master performs validation of user-submitted mapper, reducer, and other task-related classes before launching containers.
+This mechanism protects the cluster from running disallowed or unsafe task implementations as defined by administrator-controlled policies.
+- Property name: mapreduce.security.enabled
+- Property type: boolean
+- Default: false (security disabled)
+
+
+#### MapReduce Task-Level Security Enforcement: Property Domain
+Defines the set of MapReduce configuration keys that represent user-supplied class names involved in task execution (e.g., mapper, reducer, partitioner).
+The Application Master examines the values of these properties and checks whether any referenced class is listed in denied tasks.
+Administrators may override this list to expand or restrict the validation domain.
+- Property name: mapreduce.security.property-domain
+- Property type: list of configuration keys
+- Default:
+ - mapreduce.job.combine.class
+ - mapreduce.job.combiner.group.comparator.class
+ - mapreduce.job.end-notification.custom-notifier-class
+ - mapreduce.job.inputformat.class
+ - mapreduce.job.map.class
+ - mapreduce.job.map.output.collector.class
+ - mapreduce.job.output.group.comparator.class
+ - mapreduce.job.output.key.class
+ - mapreduce.job.output.key.comparator.class
+ - mapreduce.job.output.value.class
+ - mapreduce.job.outputformat.class
+ - mapreduce.job.partitioner.class
+ - mapreduce.job.reduce.class
+ - mapreduce.map.output.key.class
+ - mapreduce.map.output.value.class
Review Comment:
mapreduce.outputcommitter.factory.class
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/security/authorize/TaskLevelSecurityEnforcer.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapreduce.v2.app.security.authorize;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.MRConfig;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+
+/**
+ * Enforces task-level security rules for MapReduce jobs.
+ *
+ * <p>This security enforcement mechanism validates whether the user who submitted
+ * a job is allowed to execute the mapper/reducer/task classes defined in the job
+ * configuration. The check is performed inside the Application Master before
+ * task containers are launched.</p>
+ * <p>If the user is not on the allowed list and any job property within the configured
+ * security property domain references a denied class/prefix, a
+ * {@link TaskLevelSecurityException} is thrown and the job is rejected.</p>
+ * <p>This prevents unauthorized or unsafe custom code from running inside
+ * cluster containers.</p>
+ */
+public final class TaskLevelSecurityEnforcer {
+ private static final Logger LOG = LoggerFactory.getLogger(TaskLevelSecurityEnforcer.class);
+
+ /**
+ * Default constructor.
+ */
+ private TaskLevelSecurityEnforcer() {
+ }
+
+ /**
+ * Validates a MapReduce job's configuration against the cluster's task-level
+ * security policy.
+ *
+ * <p>The method performs the following steps:</p>
+ * <ol>
+ * <li>Check whether task-level security is enabled.</li>
+ * <li>Allow the job immediately if the user is on the configured allowed-users list.</li>
+ * <li>Retrieve the security property domain (list of job configuration keys to inspect).</li>
+ * <li>Retrieve the list of denied task class prefixes.</li>
+ * <li>For each domain property, check whether its value begins with any denied prefix.</li>
+ * <li>If a match is found, reject the job by throwing {@link TaskLevelSecurityException}.</li>
+ * </ol>
+ *
+ * @param conf the job configuration to validate
+ * @throws TaskLevelSecurityException if the user is not authorized to use one of the task classes
+ */
+ public static void validate(JobConf conf) throws TaskLevelSecurityException {
+ if (!conf.getBoolean(MRConfig.SECURITY_ENABLED, MRConfig.DEFAULT_SECURITY_ENABLED)) {
+ LOG.debug("The {} is disabled", MRConfig.SECURITY_ENABLED);
+ return;
+ }
+
+ String currentUser = conf.get(MRJobConfig.USER_NAME);
Review Comment:
+use getTrimmed() here and elsewhere to make sure the white space gets taken off
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java:
##########
@@ -133,5 +133,82 @@ public interface MRConfig {
boolean DEFAULT_MASTER_WEBAPP_UI_ACTIONS_ENABLED = true;
String MULTIPLE_OUTPUTS_CLOSE_THREAD_COUNT = "mapreduce.multiple-outputs-close-threads";
int DEFAULT_MULTIPLE_OUTPUTS_CLOSE_THREAD_COUNT = 10;
+
+ /**
+ * Enables MapReduce Task-Level Security Enforcement.
+ *
+ * When enabled, the Application Master performs validation of user-submitted
+ * mapper, reducer, and other task-related classes before launching containers.
+ * This mechanism protects the cluster from running disallowed or unsafe task
+ * implementations as defined by administrator-controlled policies.
+ *
+ * Property type: boolean
+ * Default: false (security disabled)
+ */
+ String SECURITY_ENABLED = "mapreduce.security.enabled";
+ boolean DEFAULT_SECURITY_ENABLED = false;
+
+ /**
+ * MapReduce Task-Level Security Enforcement: Property Domain
+ *
+ * Defines the set of MapReduce configuration keys that represent user-supplied
+ * class names involved in task execution (e.g., mapper, reducer, partitioner).
+ * The Application Master examines the values of these properties and checks
+ * whether any referenced class is listed in {@link #SECURITY_DENIED_TASKS}.
+ * Administrators may override this list to expand or restrict the validation
+ * domain.
+ *
+ * Property type: list of configuration keys
+ * Default: all known task-level class properties (see list below)
+ */
+ String SECURITY_PROPERTY_DOMAIN = "mapreduce.security.property-domain";
+ String[] DEFAULT_SECURITY_PROPERTY_DOMAIN = {
+ "mapreduce.job.combine.class",
+ "mapreduce.job.combiner.group.comparator.class",
+ "mapreduce.job.end-notification.custom-notifier-class",
+ "mapreduce.job.inputformat.class",
+ "mapreduce.job.map.class",
+ "mapreduce.job.map.output.collector.class",
+ "mapreduce.job.output.group.comparator.class",
+ "mapreduce.job.output.key.class",
+ "mapreduce.job.output.key.comparator.class",
+ "mapreduce.job.output.value.class",
+ "mapreduce.job.outputformat.class",
+ "mapreduce.job.partitioner.class",
+ "mapreduce.job.reduce.class",
+ "mapreduce.map.output.key.class",
+ "mapreduce.map.output.value.class"
+ };
+
+ /**
+ * MapReduce Task-Level Security Enforcement: Denied Tasks
+ *
+ * Specifies the list of disallowed task implementation classes or packages.
+ * If a user submits a job whose mapper, reducer, or other task-related classes
+ * match any entry in this blacklist.
+ *
+ * Property type: list of class name or package patterns
+ * Default: empty (no restrictions)
+ * Example: org.apache.hadoop.streaming,org.apache.hadoop.examples.QuasiMonteCarlo
+ */
+ String SECURITY_DENIED_TASKS = "mapreduce.security.denied-tasks";
+ String[] DEFAULT_SECURITY_DENIED_TASKS = {};
+
+ /**
+ * MapReduce Task-Level Security Enforcement: Allowed Users
+ *
+ * Specifies users who may bypass the blacklist defined in
+ * {@link #SECURITY_DENIED_TASKS}.
+ * This whitelist is intended for trusted or system-level workflows that may
+ * legitimately require the use of restricted task implementations.
+ * If the submitting user is listed here, blacklist enforcement is skipped,
+ * although standard Hadoop authentication and ACL checks still apply.
+ *
+ * Property type: list of usernames
+ * Default: empty (no bypass users)
+ * Example: hue,hive
+ */
+ String SECURITY_ALLOWED_USERS = "mapreduce.security.allowed-users";
Review Comment:
same comment on <p> and {@value entry}
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java:
##########
@@ -133,5 +133,82 @@ public interface MRConfig {
boolean DEFAULT_MASTER_WEBAPP_UI_ACTIONS_ENABLED = true;
String MULTIPLE_OUTPUTS_CLOSE_THREAD_COUNT = "mapreduce.multiple-outputs-close-threads";
int DEFAULT_MULTIPLE_OUTPUTS_CLOSE_THREAD_COUNT = 10;
+
+ /**
+ * Enables MapReduce Task-Level Security Enforcement.
+ *
+ * When enabled, the Application Master performs validation of user-submitted
+ * mapper, reducer, and other task-related classes before launching containers.
+ * This mechanism protects the cluster from running disallowed or unsafe task
+ * implementations as defined by administrator-controlled policies.
+ *
+ * Property type: boolean
+ * Default: false (security disabled)
Review Comment:
add a line
```
value: {@value}
```
so that the javadocs fill in the value
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java:
##########
@@ -133,5 +133,82 @@ public interface MRConfig {
boolean DEFAULT_MASTER_WEBAPP_UI_ACTIONS_ENABLED = true;
String MULTIPLE_OUTPUTS_CLOSE_THREAD_COUNT = "mapreduce.multiple-outputs-close-threads";
int DEFAULT_MULTIPLE_OUTPUTS_CLOSE_THREAD_COUNT = 10;
+
+ /**
+ * Enables MapReduce Task-Level Security Enforcement.
+ *
Review Comment:
nit, add <p> to put in newlines; needed for real javadocs, includign those IDEs can pop up
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/security/authorize/TaskLevelSecurityEnforcer.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapreduce.v2.app.security.authorize;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.MRConfig;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+
+/**
+ * Enforces task-level security rules for MapReduce jobs.
+ *
+ * <p>This security enforcement mechanism validates whether the user who submitted
+ * a job is allowed to execute the mapper/reducer/task classes defined in the job
+ * configuration. The check is performed inside the Application Master before
+ * task containers are launched.</p>
+ * <p>If the user is not on the allowed list and any job property within the configured
+ * security property domain references a denied class/prefix, a
+ * {@link TaskLevelSecurityException} is thrown and the job is rejected.</p>
+ * <p>This prevents unauthorized or unsafe custom code from running inside
+ * cluster containers.</p>
+ */
+public final class TaskLevelSecurityEnforcer {
+ private static final Logger LOG = LoggerFactory.getLogger(TaskLevelSecurityEnforcer.class);
+
+ /**
+ * Default constructor.
+ */
+ private TaskLevelSecurityEnforcer() {
+ }
+
+ /**
+ * Validates a MapReduce job's configuration against the cluster's task-level
+ * security policy.
+ *
+ * <p>The method performs the following steps:</p>
+ * <ol>
+ * <li>Check whether task-level security is enabled.</li>
+ * <li>Allow the job immediately if the user is on the configured allowed-users list.</li>
+ * <li>Retrieve the security property domain (list of job configuration keys to inspect).</li>
+ * <li>Retrieve the list of denied task class prefixes.</li>
+ * <li>For each domain property, check whether its value begins with any denied prefix.</li>
+ * <li>If a match is found, reject the job by throwing {@link TaskLevelSecurityException}.</li>
+ * </ol>
+ *
+ * @param conf the job configuration to validate
+ * @throws TaskLevelSecurityException if the user is not authorized to use one of the task classes
+ */
+ public static void validate(JobConf conf) throws TaskLevelSecurityException {
+ if (!conf.getBoolean(MRConfig.SECURITY_ENABLED, MRConfig.DEFAULT_SECURITY_ENABLED)) {
+ LOG.debug("The {} is disabled", MRConfig.SECURITY_ENABLED);
+ return;
+ }
+
+ String currentUser = conf.get(MRJobConfig.USER_NAME);
+ List<String> allowedUsers = Arrays.asList(conf.getTrimmedStrings(
+ MRConfig.SECURITY_ALLOWED_USERS,
+ MRConfig.DEFAULT_SECURITY_ALLOWED_USERS
+ ));
+ if (allowedUsers.contains(currentUser)) {
+ LOG.debug("The {} is allowed to execute every task", currentUser);
+ return;
+ }
+
+ String[] propertyDomain = conf.getTrimmedStrings(
+ MRConfig.SECURITY_PROPERTY_DOMAIN,
+ MRConfig.DEFAULT_SECURITY_PROPERTY_DOMAIN
+ );
+ String[] deniedTasks = conf.getTrimmedStrings(
+ MRConfig.SECURITY_DENIED_TASKS,
+ MRConfig.DEFAULT_SECURITY_DENIED_TASKS
+ );
+ for (String property : propertyDomain) {
+ String propertyValue = conf.get(property, "");
Review Comment:
getTrimmed
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java:
##########
@@ -133,5 +133,82 @@ public interface MRConfig {
boolean DEFAULT_MASTER_WEBAPP_UI_ACTIONS_ENABLED = true;
String MULTIPLE_OUTPUTS_CLOSE_THREAD_COUNT = "mapreduce.multiple-outputs-close-threads";
int DEFAULT_MULTIPLE_OUTPUTS_CLOSE_THREAD_COUNT = 10;
+
+ /**
+ * Enables MapReduce Task-Level Security Enforcement.
+ *
+ * When enabled, the Application Master performs validation of user-submitted
+ * mapper, reducer, and other task-related classes before launching containers.
+ * This mechanism protects the cluster from running disallowed or unsafe task
+ * implementations as defined by administrator-controlled policies.
+ *
+ * Property type: boolean
+ * Default: false (security disabled)
+ */
+ String SECURITY_ENABLED = "mapreduce.security.enabled";
+ boolean DEFAULT_SECURITY_ENABLED = false;
+
+ /**
+ * MapReduce Task-Level Security Enforcement: Property Domain
+ *
+ * Defines the set of MapReduce configuration keys that represent user-supplied
+ * class names involved in task execution (e.g., mapper, reducer, partitioner).
+ * The Application Master examines the values of these properties and checks
+ * whether any referenced class is listed in {@link #SECURITY_DENIED_TASKS}.
+ * Administrators may override this list to expand or restrict the validation
+ * domain.
+ *
+ * Property type: list of configuration keys
+ * Default: all known task-level class properties (see list below)
+ */
+ String SECURITY_PROPERTY_DOMAIN = "mapreduce.security.property-domain";
+ String[] DEFAULT_SECURITY_PROPERTY_DOMAIN = {
+ "mapreduce.job.combine.class",
+ "mapreduce.job.combiner.group.comparator.class",
+ "mapreduce.job.end-notification.custom-notifier-class",
+ "mapreduce.job.inputformat.class",
+ "mapreduce.job.map.class",
+ "mapreduce.job.map.output.collector.class",
+ "mapreduce.job.output.group.comparator.class",
+ "mapreduce.job.output.key.class",
+ "mapreduce.job.output.key.comparator.class",
+ "mapreduce.job.output.value.class",
+ "mapreduce.job.outputformat.class",
+ "mapreduce.job.partitioner.class",
+ "mapreduce.job.reduce.class",
+ "mapreduce.map.output.key.class",
+ "mapreduce.map.output.value.class"
+ };
+
+ /**
+ * MapReduce Task-Level Security Enforcement: Denied Tasks
+ *
Review Comment:
same comment on <p> and {@value entry}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-3567035442
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 26m 9s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 8m 32s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 31s | | trunk passed |
| +1 :green_heart: | compile | 18m 47s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 19m 2s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 2m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 16s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 9s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +0 :ok: | spotbugs | 0m 42s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 1m 48s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/3/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 178 extant spotbugs warnings. |
| -1 :x: | spotbugs | 1m 19s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/3/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-warnings.html) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in trunk has 39 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 29m 0s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 17m 58s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 17m 58s | | the patch passed |
| +1 :green_heart: | compile | 19m 9s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 19m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 2m 49s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/3/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 101 unchanged - 0 fixed = 103 total (was 101) |
| +1 :green_heart: | mvnsite | 2m 34s | | the patch passed |
| -1 :x: | javadoc | 0m 49s | [/results-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/3/artifact/out/results-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 1 new + 1124 unchanged - 0 fixed = 1125 total (was 1124) |
| +1 :green_heart: | javadoc | 2m 9s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +0 :ok: | spotbugs | 0m 37s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 28m 37s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 36s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 8m 34s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 8m 30s | | hadoop-mapreduce-client-app in the patch passed. |
| -1 :x: | asflicense | 1m 0s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/3/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 250m 33s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8100 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux e6a451f16c04 5.15.0-153-generic #163-Ubuntu SMP Thu Aug 7 16:37:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8c688bb887180d269effe183bffd9dc576c5b42e |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/3/testReport/ |
| Max. process+thread count | 1586 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/3/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-4120812323
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 5s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 54m 1s | | trunk passed |
| +1 :green_heart: | compile | 19m 10s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | compile | 18m 47s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | checkstyle | 5m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 22s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 2m 15s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +0 :ok: | spotbugs | 0m 43s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 0m 56s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/4/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt) | hadoop-mapreduce-client-app in trunk failed. |
| -1 :x: | spotbugs | 1m 4s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/4/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in trunk failed. |
| +1 :green_heart: | shadedclient | 35m 51s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 17m 52s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javac | 17m 52s | | the patch passed |
| +1 :green_heart: | compile | 19m 9s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | javac | 19m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 5m 45s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 14s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 2m 17s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +0 :ok: | spotbugs | 0m 40s | | hadoop-project has no data from spotbugs |
| -1 :x: | spotbugs | 1m 6s | [/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/4/artifact/out/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | spotbugs | 0m 56s | [/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/4/artifact/out/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt) | hadoop-mapreduce-client-app in the patch failed. |
| +1 :green_heart: | shadedclient | 35m 41s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 37s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 9m 14s | | hadoop-mapreduce-client-core in the patch passed. |
| -1 :x: | unit | 9m 4s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/4/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt) | hadoop-mapreduce-client-app in the patch passed. |
| +1 :green_heart: | asflicense | 1m 12s | | The patch does not generate ASF License warnings. |
| | | 266m 49s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.mapreduce.v2.app.security.authorize.TestTaskLevelSecurityEnforcer |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8100 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux 44724100a872 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e5c4a90c21ab97f1758a5342a3587bd68e51f19d |
| Default Java | Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/4/testReport/ |
| Max. process+thread count | 1089 (vs. ulimit of 10000) |
| modules | C: hadoop-project hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/4/console |
| versions | git=2.43.0 maven=3.9.11 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2555850135
##########
hadoop-project/src/site/site.xml:
##########
@@ -116,6 +116,7 @@
<item name="Pluggable Shuffle/Sort" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html"/>
<item name="Distributed Cache Deploy" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html"/>
<item name="Support for YARN Shared Cache" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html"/>
+ <item name="Task Level Security Enforcement" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/TaskLevelSecurityEnforcement.html"/>
Review Comment:
I beleive the md file will be compiled to html.
Can you find the file `SharedCacheSupport.html` above this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2974857947
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/security/authorize/TaskLevelSecurityEnforcer.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapreduce.v2.app.security.authorize;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.MRConfig;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+
+/**
+ * Enforces task-level security rules for MapReduce jobs.
+ *
+ * <p>This security enforcement mechanism validates whether the user who submitted
+ * a job is allowed to execute the mapper/reducer/task classes defined in the job
+ * configuration. The check is performed inside the Application Master before
+ * task containers are launched.</p>
+ * <p>If the user is not on the allowed list and any job property within the configured
+ * security property domain references a denied class/prefix, a
+ * {@link TaskLevelSecurityException} is thrown and the job is rejected.</p>
+ * <p>This prevents unauthorized or unsafe custom code from running inside
+ * cluster containers.</p>
+ */
+public final class TaskLevelSecurityEnforcer {
+ private static final Logger LOG = LoggerFactory.getLogger(TaskLevelSecurityEnforcer.class);
+
+ /**
+ * Default constructor.
+ */
+ private TaskLevelSecurityEnforcer() {
+ }
+
+ /**
+ * Validates a MapReduce job's configuration against the cluster's task-level
+ * security policy.
+ *
+ * <p>The method performs the following steps:</p>
+ * <ol>
+ * <li>Check whether task-level security is enabled.</li>
+ * <li>Allow the job immediately if the user is on the configured allowed-users list.</li>
+ * <li>Retrieve the security property domain (list of job configuration keys to inspect).</li>
+ * <li>Retrieve the list of denied task class prefixes.</li>
+ * <li>For each domain property, check whether its value begins with any denied prefix.</li>
+ * <li>If a match is found, reject the job by throwing {@link TaskLevelSecurityException}.</li>
+ * </ol>
+ *
+ * @param conf the job configuration to validate
+ * @throws TaskLevelSecurityException if the user is not authorized to use one of the task classes
+ */
+ public static void validate(JobConf conf) throws TaskLevelSecurityException {
+ if (!conf.getBoolean(MRConfig.SECURITY_ENABLED, MRConfig.DEFAULT_SECURITY_ENABLED)) {
+ LOG.debug("The {} is disabled", MRConfig.SECURITY_ENABLED);
+ return;
+ }
+
+ String currentUser = conf.get(MRJobConfig.USER_NAME);
+ List<String> allowedUsers = Arrays.asList(conf.getTrimmedStrings(
+ MRConfig.SECURITY_ALLOWED_USERS,
+ MRConfig.DEFAULT_SECURITY_ALLOWED_USERS
+ ));
+ if (allowedUsers.contains(currentUser)) {
+ LOG.debug("The {} is allowed to execute every task", currentUser);
+ return;
+ }
+
+ String[] propertyDomain = conf.getTrimmedStrings(
+ MRConfig.SECURITY_PROPERTY_DOMAIN,
Review Comment:
somehow like this, but on a prod cluster
https://github.com/apache/hadoop/pull/8100#pullrequestreview-3563471647
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-4112843717
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 15s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 54m 12s | | trunk passed |
| +1 :green_heart: | compile | 20m 58s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | compile | 19m 25s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | checkstyle | 6m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 18s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 2m 12s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +0 :ok: | spotbugs | 0m 43s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 0m 56s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/3/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt) | hadoop-mapreduce-client-app in trunk failed. |
| -1 :x: | spotbugs | 1m 4s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/3/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in trunk failed. |
| +1 :green_heart: | shadedclient | 34m 56s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 20m 2s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javac | 20m 2s | | the patch passed |
| +1 :green_heart: | compile | 19m 2s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | javac | 19m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 6m 5s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 43s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 11s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 2m 11s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +0 :ok: | spotbugs | 0m 38s | | hadoop-project has no data from spotbugs |
| -1 :x: | spotbugs | 1m 2s | [/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/3/artifact/out/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | spotbugs | 0m 56s | [/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/3/artifact/out/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt) | hadoop-mapreduce-client-app in the patch failed. |
| +1 :green_heart: | shadedclient | 33m 34s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 39s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 9m 5s | | hadoop-mapreduce-client-core in the patch passed. |
| -1 :x: | unit | 8m 48s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/3/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt) | hadoop-mapreduce-client-app in the patch passed. |
| +1 :green_heart: | asflicense | 1m 13s | | The patch does not generate ASF License warnings. |
| | | 267m 44s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.mapreduce.v2.app.security.authorize.TestTaskLevelSecurityEnforcer |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8100 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux 67e1a874bd50 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e5c4a90c21ab97f1758a5342a3587bd68e51f19d |
| Default Java | Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/3/testReport/ |
| Max. process+thread count | 1089 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/3/console |
| versions | git=2.43.0 maven=3.9.11 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2974992667
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/security/authorize/TaskLevelSecurityEnforcer.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapreduce.v2.app.security.authorize;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.MRConfig;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+
+/**
+ * Enforces task-level security rules for MapReduce jobs.
+ *
+ * <p>This security enforcement mechanism validates whether the user who submitted
+ * a job is allowed to execute the mapper/reducer/task classes defined in the job
+ * configuration. The check is performed inside the Application Master before
+ * task containers are launched.</p>
+ * <p>If the user is not on the allowed list and any job property within the configured
+ * security property domain references a denied class/prefix, a
+ * {@link TaskLevelSecurityException} is thrown and the job is rejected.</p>
+ * <p>This prevents unauthorized or unsafe custom code from running inside
+ * cluster containers.</p>
+ */
+public final class TaskLevelSecurityEnforcer {
+ private static final Logger LOG = LoggerFactory.getLogger(TaskLevelSecurityEnforcer.class);
+
+ /**
+ * Default constructor.
+ */
+ private TaskLevelSecurityEnforcer() {
+ }
+
+ /**
+ * Validates a MapReduce job's configuration against the cluster's task-level
+ * security policy.
+ *
+ * <p>The method performs the following steps:</p>
+ * <ol>
+ * <li>Check whether task-level security is enabled.</li>
+ * <li>Allow the job immediately if the user is on the configured allowed-users list.</li>
+ * <li>Retrieve the security property domain (list of job configuration keys to inspect).</li>
+ * <li>Retrieve the list of denied task class prefixes.</li>
+ * <li>For each domain property, check whether its value begins with any denied prefix.</li>
+ * <li>If a match is found, reject the job by throwing {@link TaskLevelSecurityException}.</li>
+ * </ol>
+ *
+ * @param conf the job configuration to validate
+ * @throws TaskLevelSecurityException if the user is not authorized to use one of the task classes
+ */
+ public static void validate(JobConf conf) throws TaskLevelSecurityException {
+ if (!conf.getBoolean(MRConfig.SECURITY_ENABLED, MRConfig.DEFAULT_SECURITY_ENABLED)) {
+ LOG.debug("The {} is disabled", MRConfig.SECURITY_ENABLED);
+ return;
+ }
+
+ String currentUser = conf.get(MRJobConfig.USER_NAME);
Review Comment:
I see your point ...
I think shortname will be fine here, cause that will be more platform independent.
If we check how `yarn.admin.acl` than we can see it works shortname also:
https://github.com/apache/hadoop/blob/23bf6c72071782e3fd5a628e21495d6b974c7a9e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/AdminACLsManager.java#L71
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-4111697368
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 19m 33s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 50m 26s | | trunk passed |
| +1 :green_heart: | compile | 18m 3s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | compile | 18m 23s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | checkstyle | 5m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 19s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 2m 13s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +0 :ok: | spotbugs | 0m 41s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 0m 56s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/1/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt) | hadoop-mapreduce-client-app in trunk failed. |
| -1 :x: | spotbugs | 1m 4s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/1/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in trunk failed. |
| +1 :green_heart: | shadedclient | 35m 9s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 17m 48s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javac | 17m 48s | | the patch passed |
| +1 :green_heart: | compile | 19m 38s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | javac | 19m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 5m 49s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 46s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 14s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 2m 14s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +0 :ok: | spotbugs | 0m 42s | | hadoop-project has no data from spotbugs |
| -1 :x: | spotbugs | 1m 7s | [/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/1/artifact/out/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt) | hadoop-mapreduce-client-core in the patch failed. |
| -1 :x: | spotbugs | 0m 58s | [/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/1/artifact/out/patch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt) | hadoop-mapreduce-client-app in the patch failed. |
| +1 :green_heart: | shadedclient | 34m 1s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 40s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 9m 15s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 8m 54s | | hadoop-mapreduce-client-app in the patch passed. |
| +1 :green_heart: | asflicense | 1m 12s | | The patch does not generate ASF License warnings. |
| | | 277m 3s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8100 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux 6296e265c5c6 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b7172f1a78243b1ae28248dc03520108be4d08fa |
| Default Java | Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/1/testReport/ |
| Max. process+thread count | 1071 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/1/console |
| versions | git=2.43.0 maven=3.9.11 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-3566978742
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 8m 10s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 15m 48s | | trunk passed |
| +1 :green_heart: | compile | 8m 47s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 54s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 1m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 13s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +0 :ok: | spotbugs | 0m 24s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 0m 55s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/4/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 178 extant spotbugs warnings. |
| -1 :x: | spotbugs | 0m 43s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/4/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-warnings.html) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in trunk has 39 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 14m 40s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 17s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 48s | | the patch passed |
| +1 :green_heart: | compile | 8m 24s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 8m 24s | | the patch passed |
| +1 :green_heart: | compile | 8m 52s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 8m 52s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 38s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/4/artifact/out/results-checkstyle-root.txt) | root: The patch generated 4 new + 101 unchanged - 0 fixed = 105 total (was 101) |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| -1 :x: | javadoc | 0m 26s | [/results-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/4/artifact/out/results-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 1 new + 1124 unchanged - 0 fixed = 1125 total (was 1124) |
| +1 :green_heart: | javadoc | 1m 8s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +0 :ok: | spotbugs | 0m 20s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 14m 42s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 19s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 5m 56s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 5m 55s | | hadoop-mapreduce-client-app in the patch passed. |
| -1 :x: | asflicense | 0m 34s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/4/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 121m 8s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8100 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux 1bc164bf8b7d 5.15.0-153-generic #163-Ubuntu SMP Thu Aug 7 16:37:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 018e4cb99a75e9aefe43333abddddaae9c2eef1d |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/4/testReport/ |
| Max. process+thread count | 1592 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/4/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2555861049
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/security/authorize/TestTaskLevelSecurityEnforcer.java:
##########
@@ -0,0 +1,140 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapreduce.v2.app.security.authorize;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.MRConfig;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class TestTaskLevelSecurityEnforcer {
Review Comment:
I believe if we use a common conf object, than we wont be able to run the UT`s parallel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2559470037
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/security/authorize/TestTaskLevelSecurityEnforcer.java:
##########
@@ -0,0 +1,140 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapreduce.v2.app.security.authorize;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.MRConfig;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class TestTaskLevelSecurityEnforcer {
Review Comment:
extends 'AbstractHadoopTestBase`
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/security/authorize/TaskLevelSecurityEnforcer.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapreduce.v2.app.security.authorize;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.MRConfig;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+
+/**
+ * Enforces task-level security rules for MapReduce jobs.
+ *
+ * <p>This security enforcement mechanism validates whether the user who submitted
+ * a job is allowed to execute the mapper/reducer/task classes defined in the job
+ * configuration. The check is performed inside the Application Master before
+ * task containers are launched.</p>
+ * <p>If the user is not on the allowed list and any job property within the configured
+ * security property domain references a denied class/prefix, a
+ * {@link TaskLevelSecurityException} is thrown and the job is rejected.</p>
+ * <p>This prevents unauthorized or unsafe custom code from running inside
+ * cluster containers.</p>
+ */
+public final class TaskLevelSecurityEnforcer {
+ private static final Logger LOG = LoggerFactory.getLogger(TaskLevelSecurityEnforcer.class);
+
+ /**
+ * Default constructor.
+ */
+ private TaskLevelSecurityEnforcer() {
+ }
+
+ /**
+ * Validates a MapReduce job's configuration against the cluster's task-level
+ * security policy.
+ *
+ * <p>The method performs the following steps:</p>
+ * <ol>
+ * <li>Check whether task-level security is enabled.</li>
+ * <li>Allow the job immediately if the user is on the configured allowed-users list.</li>
+ * <li>Retrieve the security property domain (list of job configuration keys to inspect).</li>
+ * <li>Retrieve the list of denied task class prefixes.</li>
+ * <li>For each domain property, check whether its value begins with any denied prefix.</li>
+ * <li>If a match is found, reject the job by throwing {@link TaskLevelSecurityException}.</li>
+ * </ol>
+ *
+ * @param conf the job configuration to validate
+ * @throws TaskLevelSecurityException if the user is not authorized to use one of the task classes
+ */
+ public static void validate(JobConf conf) throws TaskLevelSecurityException {
+ if (!conf.getBoolean(MRConfig.SECURITY_ENABLED, MRConfig.DEFAULT_SECURITY_ENABLED)) {
+ LOG.debug("The {} is disabled", MRConfig.SECURITY_ENABLED);
+ return;
+ }
+
+ String currentUser = conf.get(MRJobConfig.USER_NAME);
Review Comment:
looking at JobSubmitter, this is set to UGI.currentUser.shortName. Is that enough? Shouldn't it be on fullName?
if so, JobSubmitter should add that as a property too
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml:
##########
@@ -2282,4 +2282,44 @@
</description>
</property>
+<property>
+ <name>mapreduce.security.enabled</name>
+ <value>false</value>
+ <description>
+ Enables MapReduce Task-Level Security Enforcement
+ When enabled, the Application Master performs validation of user-submitted mapper, reducer, and other task-related classes before launching containers.
+ This mechanism protects the cluster from running disallowed or unsafe task implementations as defined by administrator-controlled policies.
+ </description>
+</property>
+
+<property>
+ <name>mapreduce.security.property-domain</name>
+ <value>mapreduce.job.combine.class,mapreduce.job.combiner.group.comparator.class,mapreduce.job.end-notification.custom-notifier-class,mapreduce.job.inputformat.class,mapreduce.job.map.class,mapreduce.job.map.output.collector.class,mapreduce.job.output.group.comparator.class,mapreduce.job.output.key.class,mapreduce.job.output.key.comparator.class,mapreduce.job.output.value.class,mapreduce.job.outputformat.class,mapreduce.job.partitioner.class,mapreduce.job.reduce.class,mapreduce.map.output.key.class,mapreduce.map.output.value.class</value>
+ <description>
Review Comment:
add all the chaining task properties too, presumably
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "zeekling (via GitHub)" <gi...@apache.org>.
zeekling commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2596299848
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/security/authorize/TaskLevelSecurityEnforcer.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapreduce.v2.app.security.authorize;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.MRConfig;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+
+/**
+ * Enforces task-level security rules for MapReduce jobs.
+ *
+ * <p>This security enforcement mechanism validates whether the user who submitted
+ * a job is allowed to execute the mapper/reducer/task classes defined in the job
+ * configuration. The check is performed inside the Application Master before
+ * task containers are launched.</p>
+ * <p>If the user is not on the allowed list and any job property within the configured
+ * security property domain references a denied class/prefix, a
+ * {@link TaskLevelSecurityException} is thrown and the job is rejected.</p>
+ * <p>This prevents unauthorized or unsafe custom code from running inside
+ * cluster containers.</p>
+ */
+public final class TaskLevelSecurityEnforcer {
+ private static final Logger LOG = LoggerFactory.getLogger(TaskLevelSecurityEnforcer.class);
+
+ /**
+ * Default constructor.
+ */
+ private TaskLevelSecurityEnforcer() {
+ }
+
+ /**
+ * Validates a MapReduce job's configuration against the cluster's task-level
+ * security policy.
+ *
+ * <p>The method performs the following steps:</p>
+ * <ol>
+ * <li>Check whether task-level security is enabled.</li>
+ * <li>Allow the job immediately if the user is on the configured allowed-users list.</li>
+ * <li>Retrieve the security property domain (list of job configuration keys to inspect).</li>
+ * <li>Retrieve the list of denied task class prefixes.</li>
+ * <li>For each domain property, check whether its value begins with any denied prefix.</li>
+ * <li>If a match is found, reject the job by throwing {@link TaskLevelSecurityException}.</li>
+ * </ol>
+ *
+ * @param conf the job configuration to validate
+ * @throws TaskLevelSecurityException if the user is not authorized to use one of the task classes
+ */
+ public static void validate(JobConf conf) throws TaskLevelSecurityException {
+ if (!conf.getBoolean(MRConfig.SECURITY_ENABLED, MRConfig.DEFAULT_SECURITY_ENABLED)) {
+ LOG.debug("The {} is disabled", MRConfig.SECURITY_ENABLED);
+ return;
+ }
+
+ String currentUser = conf.get(MRJobConfig.USER_NAME);
+ List<String> allowedUsers = Arrays.asList(conf.getTrimmedStrings(
+ MRConfig.SECURITY_ALLOWED_USERS,
+ MRConfig.DEFAULT_SECURITY_ALLOWED_USERS
+ ));
+ if (allowedUsers.contains(currentUser)) {
+ LOG.debug("The {} is allowed to execute every task", currentUser);
+ return;
+ }
+
+ String[] propertyDomain = conf.getTrimmedStrings(
+ MRConfig.SECURITY_PROPERTY_DOMAIN,
+ MRConfig.DEFAULT_SECURITY_PROPERTY_DOMAIN
+ );
+ String[] deniedTasks = conf.getTrimmedStrings(
+ MRConfig.SECURITY_DENIED_TASKS,
+ MRConfig.DEFAULT_SECURITY_DENIED_TASKS
+ );
+ for (String property : propertyDomain) {
+ String propertyValue = conf.get(property, "");
+ for (String deniedTask : deniedTasks) {
+ if (propertyValue.startsWith(deniedTask)) {
Review Comment:
It is good to use equals
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/security/authorize/TaskLevelSecurityEnforcer.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapreduce.v2.app.security.authorize;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.MRConfig;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+
+/**
+ * Enforces task-level security rules for MapReduce jobs.
+ *
+ * <p>This security enforcement mechanism validates whether the user who submitted
+ * a job is allowed to execute the mapper/reducer/task classes defined in the job
+ * configuration. The check is performed inside the Application Master before
+ * task containers are launched.</p>
+ * <p>If the user is not on the allowed list and any job property within the configured
+ * security property domain references a denied class/prefix, a
+ * {@link TaskLevelSecurityException} is thrown and the job is rejected.</p>
+ * <p>This prevents unauthorized or unsafe custom code from running inside
+ * cluster containers.</p>
+ */
+public final class TaskLevelSecurityEnforcer {
+ private static final Logger LOG = LoggerFactory.getLogger(TaskLevelSecurityEnforcer.class);
+
+ /**
+ * Default constructor.
+ */
+ private TaskLevelSecurityEnforcer() {
+ }
+
+ /**
+ * Validates a MapReduce job's configuration against the cluster's task-level
+ * security policy.
+ *
+ * <p>The method performs the following steps:</p>
+ * <ol>
+ * <li>Check whether task-level security is enabled.</li>
+ * <li>Allow the job immediately if the user is on the configured allowed-users list.</li>
+ * <li>Retrieve the security property domain (list of job configuration keys to inspect).</li>
+ * <li>Retrieve the list of denied task class prefixes.</li>
+ * <li>For each domain property, check whether its value begins with any denied prefix.</li>
+ * <li>If a match is found, reject the job by throwing {@link TaskLevelSecurityException}.</li>
+ * </ol>
+ *
+ * @param conf the job configuration to validate
+ * @throws TaskLevelSecurityException if the user is not authorized to use one of the task classes
+ */
+ public static void validate(JobConf conf) throws TaskLevelSecurityException {
+ if (!conf.getBoolean(MRConfig.SECURITY_ENABLED, MRConfig.DEFAULT_SECURITY_ENABLED)) {
+ LOG.debug("The {} is disabled", MRConfig.SECURITY_ENABLED);
+ return;
+ }
+
+ String currentUser = conf.get(MRJobConfig.USER_NAME);
+ List<String> allowedUsers = Arrays.asList(conf.getTrimmedStrings(
+ MRConfig.SECURITY_ALLOWED_USERS,
+ MRConfig.DEFAULT_SECURITY_ALLOWED_USERS
+ ));
+ if (allowedUsers.contains(currentUser)) {
+ LOG.debug("The {} is allowed to execute every task", currentUser);
+ return;
+ }
+
+ String[] propertyDomain = conf.getTrimmedStrings(
+ MRConfig.SECURITY_PROPERTY_DOMAIN,
Review Comment:
I don't understand what the meaning of this parameter is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-4118973067
Reopen as i do think it could be useful -there's some outstanding issues to get in
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-4126033609
<img width="1022" height="902" alt="image" src="https://github.com/user-attachments/assets/1cc05e6e-db36-4ae5-8f49-f9d96ead69a7" />
dont think i have the permission
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #8100: MAPREDUCE-7523. MapReduce Task-Level Security Enforcement
URL: https://github.com/apache/hadoop/pull/8100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2555796877
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/security/authorize/TestTaskLevelSecurityEnforcer.java:
##########
@@ -0,0 +1,140 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapreduce.v2.app.security.authorize;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.MRConfig;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class TestTaskLevelSecurityEnforcer {
Review Comment:
What do you think of adding the set up method like the followings?
```
private JobConf conf;
@BeforeEach
public void setUp() {
conf = new JobConf();
}
```
And remove the duplicate `JobConf conf = new JobConf();` at most of the methods.
I am not sure it is a good idea or declaring it add each method is more readable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-3567084232
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 8m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 15m 40s | | trunk passed |
| +1 :green_heart: | compile | 8m 46s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 43s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 1m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 20s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +0 :ok: | spotbugs | 0m 24s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 0m 55s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/7/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 178 extant spotbugs warnings. |
| -1 :x: | spotbugs | 0m 40s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/7/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-warnings.html) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in trunk has 39 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 14m 12s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 48s | | the patch passed |
| +1 :green_heart: | compile | 8m 14s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 8m 14s | | the patch passed |
| +1 :green_heart: | compile | 8m 46s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 8m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 15s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 7s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +0 :ok: | spotbugs | 0m 20s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 14m 30s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 20s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 5m 49s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 5m 44s | | hadoop-mapreduce-client-app in the patch passed. |
| -1 :x: | asflicense | 0m 36s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/7/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 119m 42s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8100 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux f8ffa8c85047 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b7172f1a78243b1ae28248dc03520108be4d08fa |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/7/testReport/ |
| Max. process+thread count | 1592 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/7/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2556060895
##########
hadoop-project/src/site/site.xml:
##########
@@ -116,6 +116,7 @@
<item name="Pluggable Shuffle/Sort" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html"/>
<item name="Distributed Cache Deploy" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html"/>
<item name="Support for YARN Shared Cache" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html"/>
+ <item name="Task Level Security Enforcement" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/TaskLevelSecurityEnforcement.html"/>
Review Comment:
Ahh...you are right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-3566925108
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 7m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 15m 20s | | trunk passed |
| +1 :green_heart: | compile | 8m 47s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 47s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 1m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +0 :ok: | spotbugs | 0m 25s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 1m 0s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/1/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 178 extant spotbugs warnings. |
| -1 :x: | spotbugs | 0m 41s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/1/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-warnings.html) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in trunk has 39 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 15m 14s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 18s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 45s | | the patch passed |
| +1 :green_heart: | compile | 8m 16s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 8m 16s | | the patch passed |
| +1 :green_heart: | compile | 8m 45s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 8m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 31s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/1/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 101 unchanged - 0 fixed = 103 total (was 101) |
| +1 :green_heart: | mvnsite | 1m 24s | | the patch passed |
| -1 :x: | javadoc | 0m 27s | [/results-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/1/artifact/out/results-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 1 new + 1124 unchanged - 0 fixed = 1125 total (was 1124) |
| +1 :green_heart: | javadoc | 1m 10s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +0 :ok: | spotbugs | 0m 19s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 14m 51s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 21s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 5m 59s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 5m 53s | | hadoop-mapreduce-client-app in the patch passed. |
| -1 :x: | asflicense | 0m 35s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/1/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 120m 21s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8100 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux 7496e8feb77a 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 67057308f861fad793add59108f674951ac5b808 |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/1/testReport/ |
| Max. process+thread count | 1595 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/1/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-3567006298
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 7m 15s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 15m 54s | | trunk passed |
| +1 :green_heart: | compile | 8m 40s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 47s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 1m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 11s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +0 :ok: | spotbugs | 0m 25s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 1m 0s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/6/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 178 extant spotbugs warnings. |
| -1 :x: | spotbugs | 0m 41s | [/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/6/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-warnings.html) | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in trunk has 39 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 14m 23s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 18s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 45s | | the patch passed |
| +1 :green_heart: | compile | 8m 9s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 8m 9s | | the patch passed |
| +1 :green_heart: | compile | 8m 48s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 8m 48s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 30s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/6/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 101 unchanged - 0 fixed = 103 total (was 101) |
| +1 :green_heart: | mvnsite | 1m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 7s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 9s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +0 :ok: | spotbugs | 0m 19s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 14m 28s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 21s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 5m 46s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 5m 51s | | hadoop-mapreduce-client-app in the patch passed. |
| -1 :x: | asflicense | 0m 34s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/6/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 119m 3s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8100 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux ab85711b9718 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 09ca94bee28f52378a2679bdbf35e2226c7bc7a5 |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/6/testReport/ |
| Max. process+thread count | 1594 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/6/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-4101610873
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-4109851031
@K0K0V0K hoping to make progress here if you address the comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-4110574562
I just rebased the branch, i will fix the comments now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-4124062818
@steveloughran
I think this PR is getting closed cause kinda old.
If you dont mind i created a new one where we should not face with this issue:
https://github.com/apache/hadoop/pull/8378
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #8100: MAPREDUCE-7523. MapReduce Task-Level Security Enforcement
URL: https://github.com/apache/hadoop/pull/8100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2555861049
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/security/authorize/TestTaskLevelSecurityEnforcer.java:
##########
@@ -0,0 +1,140 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapreduce.v2.app.security.authorize;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.MRConfig;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class TestTaskLevelSecurityEnforcer {
Review Comment:
I believe if we use a common conf object, than we won`t be able to run the UT`s parallel.
##########
hadoop-project/src/site/site.xml:
##########
@@ -116,6 +116,7 @@
<item name="Pluggable Shuffle/Sort" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html"/>
<item name="Distributed Cache Deploy" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html"/>
<item name="Support for YARN Shared Cache" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html"/>
+ <item name="Task Level Security Enforcement" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/TaskLevelSecurityEnforcement.html"/>
Review Comment:
I believe the md file will be compiled to html.
Can you find the file `SharedCacheSupport.html` above this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2555861049
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/security/authorize/TestTaskLevelSecurityEnforcer.java:
##########
@@ -0,0 +1,140 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapreduce.v2.app.security.authorize;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.MRConfig;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class TestTaskLevelSecurityEnforcer {
Review Comment:
I believe if we use a common conf object, than we wont be able to run the UT`s parallel.
Also for me seems against the clean code:
- create a new method for one line
- create local variables, what were not present before
- use new annotation what increase the risk in a future test framework migration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2555861049
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/security/authorize/TestTaskLevelSecurityEnforcer.java:
##########
@@ -0,0 +1,140 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapreduce.v2.app.security.authorize;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.MRConfig;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class TestTaskLevelSecurityEnforcer {
Review Comment:
I believe if we use a common conf object, than we wont be able to run the UT`s parallel.
Also for me seems against the clean code:
- create a new method for one line
- create local variables, what was not present before
- use new annotation what increase the risk in a future test framework migration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2556067462
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/security/authorize/TestTaskLevelSecurityEnforcer.java:
##########
@@ -0,0 +1,140 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapreduce.v2.app.security.authorize;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.MRConfig;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class TestTaskLevelSecurityEnforcer {
Review Comment:
It was able to run in parallel with `@BeforeEach`.
Now I agree seems like declare config at each method is more cleaner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2974854542
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/security/authorize/TaskLevelSecurityEnforcer.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapreduce.v2.app.security.authorize;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.MRConfig;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+
+/**
+ * Enforces task-level security rules for MapReduce jobs.
+ *
+ * <p>This security enforcement mechanism validates whether the user who submitted
+ * a job is allowed to execute the mapper/reducer/task classes defined in the job
+ * configuration. The check is performed inside the Application Master before
+ * task containers are launched.</p>
+ * <p>If the user is not on the allowed list and any job property within the configured
+ * security property domain references a denied class/prefix, a
+ * {@link TaskLevelSecurityException} is thrown and the job is rejected.</p>
+ * <p>This prevents unauthorized or unsafe custom code from running inside
+ * cluster containers.</p>
+ */
+public final class TaskLevelSecurityEnforcer {
+ private static final Logger LOG = LoggerFactory.getLogger(TaskLevelSecurityEnforcer.class);
+
+ /**
+ * Default constructor.
+ */
+ private TaskLevelSecurityEnforcer() {
+ }
+
+ /**
+ * Validates a MapReduce job's configuration against the cluster's task-level
+ * security policy.
+ *
+ * <p>The method performs the following steps:</p>
+ * <ol>
+ * <li>Check whether task-level security is enabled.</li>
+ * <li>Allow the job immediately if the user is on the configured allowed-users list.</li>
+ * <li>Retrieve the security property domain (list of job configuration keys to inspect).</li>
+ * <li>Retrieve the list of denied task class prefixes.</li>
+ * <li>For each domain property, check whether its value begins with any denied prefix.</li>
+ * <li>If a match is found, reject the job by throwing {@link TaskLevelSecurityException}.</li>
+ * </ol>
+ *
+ * @param conf the job configuration to validate
+ * @throws TaskLevelSecurityException if the user is not authorized to use one of the task classes
+ */
+ public static void validate(JobConf conf) throws TaskLevelSecurityException {
+ if (!conf.getBoolean(MRConfig.SECURITY_ENABLED, MRConfig.DEFAULT_SECURITY_ENABLED)) {
+ LOG.debug("The {} is disabled", MRConfig.SECURITY_ENABLED);
+ return;
+ }
+
+ String currentUser = conf.get(MRJobConfig.USER_NAME);
+ List<String> allowedUsers = Arrays.asList(conf.getTrimmedStrings(
+ MRConfig.SECURITY_ALLOWED_USERS,
+ MRConfig.DEFAULT_SECURITY_ALLOWED_USERS
+ ));
+ if (allowedUsers.contains(currentUser)) {
+ LOG.debug("The {} is allowed to execute every task", currentUser);
+ return;
+ }
+
+ String[] propertyDomain = conf.getTrimmedStrings(
+ MRConfig.SECURITY_PROPERTY_DOMAIN,
Review Comment:
In case if i forget to add something to this code, or i added something what should not be there this is a back up plan to runtime fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-4121159248
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 16s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 22s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 55m 32s | | trunk passed |
| +1 :green_heart: | compile | 18m 7s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | compile | 20m 11s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | checkstyle | 6m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 24s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 3m 14s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +0 :ok: | spotbugs | 0m 43s | | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 34m 59s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 21s | | the patch passed |
| +1 :green_heart: | compile | 17m 0s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javac | 17m 0s | | the patch passed |
| +1 :green_heart: | compile | 18m 56s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | javac | 18m 56s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 6m 5s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 48s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 18s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 2m 20s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +0 :ok: | spotbugs | 0m 42s | | hadoop-project has no data from spotbugs |
| +1 :green_heart: | shadedclient | 34m 33s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 39s | | hadoop-project in the patch passed. |
| +1 :green_heart: | unit | 8m 55s | | hadoop-mapreduce-client-core in the patch passed. |
| -1 :x: | unit | 8m 48s | [/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/5/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt) | hadoop-mapreduce-client-app in the patch passed. |
| +1 :green_heart: | asflicense | 1m 11s | | The patch does not generate ASF License warnings. |
| | | 269m 16s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.mapreduce.v2.app.security.authorize.TestTaskLevelSecurityEnforcer |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8100 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint markdownlint |
| uname | Linux b3413a0d711b 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9acade01a6d48f5ee11df92a344b27fff18301eb |
| Default Java | Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/5/testReport/ |
| Max. process+thread count | 1087 (vs. ulimit of 10000) |
| modules | C: hadoop-project hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8100/5/console |
| versions | git=2.43.0 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2974866697
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml:
##########
@@ -2282,4 +2282,44 @@
</description>
</property>
+<property>
+ <name>mapreduce.security.enabled</name>
+ <value>false</value>
+ <description>
+ Enables MapReduce Task-Level Security Enforcement
+ When enabled, the Application Master performs validation of user-submitted mapper, reducer, and other task-related classes before launching containers.
+ This mechanism protects the cluster from running disallowed or unsafe task implementations as defined by administrator-controlled policies.
+ </description>
+</property>
+
+<property>
+ <name>mapreduce.security.property-domain</name>
+ <value>mapreduce.job.combine.class,mapreduce.job.combiner.group.comparator.class,mapreduce.job.end-notification.custom-notifier-class,mapreduce.job.inputformat.class,mapreduce.job.map.class,mapreduce.job.map.output.collector.class,mapreduce.job.output.group.comparator.class,mapreduce.job.output.key.class,mapreduce.job.output.key.comparator.class,mapreduce.job.output.value.class,mapreduce.job.outputformat.class,mapreduce.job.partitioner.class,mapreduce.job.reduce.class,mapreduce.map.output.key.class,mapreduce.map.output.value.class</value>
+ <description>
Review Comment:
Sorry, i dont know what `chaining task properties` means in this context
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2975230949
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java:
##########
@@ -133,5 +133,82 @@ public interface MRConfig {
boolean DEFAULT_MASTER_WEBAPP_UI_ACTIONS_ENABLED = true;
String MULTIPLE_OUTPUTS_CLOSE_THREAD_COUNT = "mapreduce.multiple-outputs-close-threads";
int DEFAULT_MULTIPLE_OUTPUTS_CLOSE_THREAD_COUNT = 10;
+
+ /**
+ * Enables MapReduce Task-Level Security Enforcement.
+ *
+ * When enabled, the Application Master performs validation of user-submitted
+ * mapper, reducer, and other task-related classes before launching containers.
+ * This mechanism protects the cluster from running disallowed or unsafe task
+ * implementations as defined by administrator-controlled policies.
+ *
+ * Property type: boolean
+ * Default: false (security disabled)
+ */
+ String SECURITY_ENABLED = "mapreduce.security.enabled";
+ boolean DEFAULT_SECURITY_ENABLED = false;
+
+ /**
+ * MapReduce Task-Level Security Enforcement: Property Domain
+ *
+ * Defines the set of MapReduce configuration keys that represent user-supplied
+ * class names involved in task execution (e.g., mapper, reducer, partitioner).
+ * The Application Master examines the values of these properties and checks
+ * whether any referenced class is listed in {@link #SECURITY_DENIED_TASKS}.
+ * Administrators may override this list to expand or restrict the validation
+ * domain.
+ *
+ * Property type: list of configuration keys
+ * Default: all known task-level class properties (see list below)
+ */
+ String SECURITY_PROPERTY_DOMAIN = "mapreduce.security.property-domain";
+ String[] DEFAULT_SECURITY_PROPERTY_DOMAIN = {
+ "mapreduce.job.combine.class",
+ "mapreduce.job.combiner.group.comparator.class",
+ "mapreduce.job.end-notification.custom-notifier-class",
+ "mapreduce.job.inputformat.class",
+ "mapreduce.job.map.class",
+ "mapreduce.job.map.output.collector.class",
+ "mapreduce.job.output.group.comparator.class",
+ "mapreduce.job.output.key.class",
+ "mapreduce.job.output.key.comparator.class",
+ "mapreduce.job.output.value.class",
+ "mapreduce.job.outputformat.class",
+ "mapreduce.job.partitioner.class",
+ "mapreduce.job.reduce.class",
+ "mapreduce.map.output.key.class",
+ "mapreduce.map.output.value.class"
+ };
+
+ /**
+ * MapReduce Task-Level Security Enforcement: Denied Tasks
+ *
Review Comment:
<img width="992" height="386" alt="image" src="https://github.com/user-attachments/assets/4df10d95-7c03-4e1e-891a-1153adc5c97d" />
I dont think this is working for arrays, seems like only const supported
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-3580839466
First of all thanks @steveloughran for the detailed review!
> So this is a kerberized? non kerberized? cluster where you want to restrict the specific code an untrusted user may execute. Presumably you have to complete the lockdown the ability for them to add any new classes to the classpath, otherwise they would just add a new mapper or reducer class.
I tested this on a non-kerberized cluster, but you’re right, I will also verify it on a kerberized cluster.
This feature is not intended to prevent certain types of mapper/reducer code from being uploaded to the YARN cluster.
For example, a user might have an old legacy MapReduce library that cannot be removed from processes, otherwise their business could fall apart, like a Banksy artwork in a paper shredder.
The purpose of this feature is that if you must use a specific MapReduce library, at least you can restrict the user domain that has access to it, which reduces the potential attack surface.
This feature is not designed to mitigate the risk of users uploading custom mapper/reducer code.
> Is there any write-up on the security model, attacks it is intended to defend against etc?
Not yet but i will add to the docs.
> If you are trying to stop untrusted users from having their classes loaded in the cluster –I don't think it is sufficient. You would have to audit every single place where we instantiate a class through Configuration. A quick scan of Configuration.getClass use shows there are some distcp references (distcp.copy.listing.class,...) and shows up mapreduce.chain.mapper as another mechanism used to chain together tasks -it'll need lockdown too.
Yes, you’re right, this cannot prevent users from uploading their custom code. I hope this can be achieved through other security mechanisms.
> I think I will need to see the design goal of the security measure, the threat model it intends to mitigate and whether you want that mitigation to be absolute or best-effort. I'm curious about which tasks you want to lock down as well while still having them on the classpath.
I will add the goal to the TaskLevelSecurityEnforcement.md file of this PR, if OK for you
> I really think it'll be hard to stop someone sufficiently motivated from executing code in the cluster.
100% agree, this can be hardly achievable from hadoop code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2974849166
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/security/authorize/TaskLevelSecurityEnforcer.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapreduce.v2.app.security.authorize;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapreduce.MRConfig;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+
+/**
+ * Enforces task-level security rules for MapReduce jobs.
+ *
+ * <p>This security enforcement mechanism validates whether the user who submitted
+ * a job is allowed to execute the mapper/reducer/task classes defined in the job
+ * configuration. The check is performed inside the Application Master before
+ * task containers are launched.</p>
+ * <p>If the user is not on the allowed list and any job property within the configured
+ * security property domain references a denied class/prefix, a
+ * {@link TaskLevelSecurityException} is thrown and the job is rejected.</p>
+ * <p>This prevents unauthorized or unsafe custom code from running inside
+ * cluster containers.</p>
+ */
+public final class TaskLevelSecurityEnforcer {
+ private static final Logger LOG = LoggerFactory.getLogger(TaskLevelSecurityEnforcer.class);
+
+ /**
+ * Default constructor.
+ */
+ private TaskLevelSecurityEnforcer() {
+ }
+
+ /**
+ * Validates a MapReduce job's configuration against the cluster's task-level
+ * security policy.
+ *
+ * <p>The method performs the following steps:</p>
+ * <ol>
+ * <li>Check whether task-level security is enabled.</li>
+ * <li>Allow the job immediately if the user is on the configured allowed-users list.</li>
+ * <li>Retrieve the security property domain (list of job configuration keys to inspect).</li>
+ * <li>Retrieve the list of denied task class prefixes.</li>
+ * <li>For each domain property, check whether its value begins with any denied prefix.</li>
+ * <li>If a match is found, reject the job by throwing {@link TaskLevelSecurityException}.</li>
+ * </ol>
+ *
+ * @param conf the job configuration to validate
+ * @throws TaskLevelSecurityException if the user is not authorized to use one of the task classes
+ */
+ public static void validate(JobConf conf) throws TaskLevelSecurityException {
+ if (!conf.getBoolean(MRConfig.SECURITY_ENABLED, MRConfig.DEFAULT_SECURITY_ENABLED)) {
+ LOG.debug("The {} is disabled", MRConfig.SECURITY_ENABLED);
+ return;
+ }
+
+ String currentUser = conf.get(MRJobConfig.USER_NAME);
+ List<String> allowedUsers = Arrays.asList(conf.getTrimmedStrings(
+ MRConfig.SECURITY_ALLOWED_USERS,
+ MRConfig.DEFAULT_SECURITY_ALLOWED_USERS
+ ));
+ if (allowedUsers.contains(currentUser)) {
+ LOG.debug("The {} is allowed to execute every task", currentUser);
+ return;
+ }
+
+ String[] propertyDomain = conf.getTrimmedStrings(
+ MRConfig.SECURITY_PROPERTY_DOMAIN,
+ MRConfig.DEFAULT_SECURITY_PROPERTY_DOMAIN
+ );
+ String[] deniedTasks = conf.getTrimmedStrings(
+ MRConfig.SECURITY_DENIED_TASKS,
+ MRConfig.DEFAULT_SECURITY_DENIED_TASKS
+ );
+ for (String property : propertyDomain) {
+ String propertyValue = conf.get(property, "");
+ for (String deniedTask : deniedTasks) {
+ if (propertyValue.startsWith(deniedTask)) {
Review Comment:
Are you sure?
Lets say we want to deny every thing under `org.apache.hadoop.streaming` , so we set
mapreduce.security.denied-tasks=org.apache.hadoop.streaming
If the mapper in the job set to `org.apache.hadoop.streaming.PipeMapper`, this will be evaulated like
"org.apache.hadoop.streaming.PipeMapper".equals("org.apache.hadoop.streaming") what will be false, but expected value is true.
But maybe i am wrong please correct me in that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-4110307279
Hey @steveloughran !
Thanks for checking back on this!
I will address them shortly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2975230949
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java:
##########
@@ -133,5 +133,82 @@ public interface MRConfig {
boolean DEFAULT_MASTER_WEBAPP_UI_ACTIONS_ENABLED = true;
String MULTIPLE_OUTPUTS_CLOSE_THREAD_COUNT = "mapreduce.multiple-outputs-close-threads";
int DEFAULT_MULTIPLE_OUTPUTS_CLOSE_THREAD_COUNT = 10;
+
+ /**
+ * Enables MapReduce Task-Level Security Enforcement.
+ *
+ * When enabled, the Application Master performs validation of user-submitted
+ * mapper, reducer, and other task-related classes before launching containers.
+ * This mechanism protects the cluster from running disallowed or unsafe task
+ * implementations as defined by administrator-controlled policies.
+ *
+ * Property type: boolean
+ * Default: false (security disabled)
+ */
+ String SECURITY_ENABLED = "mapreduce.security.enabled";
+ boolean DEFAULT_SECURITY_ENABLED = false;
+
+ /**
+ * MapReduce Task-Level Security Enforcement: Property Domain
+ *
+ * Defines the set of MapReduce configuration keys that represent user-supplied
+ * class names involved in task execution (e.g., mapper, reducer, partitioner).
+ * The Application Master examines the values of these properties and checks
+ * whether any referenced class is listed in {@link #SECURITY_DENIED_TASKS}.
+ * Administrators may override this list to expand or restrict the validation
+ * domain.
+ *
+ * Property type: list of configuration keys
+ * Default: all known task-level class properties (see list below)
+ */
+ String SECURITY_PROPERTY_DOMAIN = "mapreduce.security.property-domain";
+ String[] DEFAULT_SECURITY_PROPERTY_DOMAIN = {
+ "mapreduce.job.combine.class",
+ "mapreduce.job.combiner.group.comparator.class",
+ "mapreduce.job.end-notification.custom-notifier-class",
+ "mapreduce.job.inputformat.class",
+ "mapreduce.job.map.class",
+ "mapreduce.job.map.output.collector.class",
+ "mapreduce.job.output.group.comparator.class",
+ "mapreduce.job.output.key.class",
+ "mapreduce.job.output.key.comparator.class",
+ "mapreduce.job.output.value.class",
+ "mapreduce.job.outputformat.class",
+ "mapreduce.job.partitioner.class",
+ "mapreduce.job.reduce.class",
+ "mapreduce.map.output.key.class",
+ "mapreduce.map.output.value.class"
+ };
+
+ /**
+ * MapReduce Task-Level Security Enforcement: Denied Tasks
+ *
Review Comment:
<img width="992" height="386" alt="image" src="https://github.com/user-attachments/assets/4df10d95-7c03-4e1e-891a-1153adc5c97d" />
I dont think this is working for arrays, seems like only const supported
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-4110881679
I tried to address the open issues based on the review.
May i ask a review again please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #8100: MAPREDUCE-7523. MapReduce Task-Level Security Enforcement
URL: https://github.com/apache/hadoop/pull/8100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#issuecomment-4125980582
could you re-open this one? I don't have that button enabled for me...wondering if you could.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] MAPREDUCE-7523. MapReduce Task-Level Security Enforcement [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8100:
URL: https://github.com/apache/hadoop/pull/8100#discussion_r2555732023
##########
hadoop-project/src/site/site.xml:
##########
@@ -116,6 +116,7 @@
<item name="Pluggable Shuffle/Sort" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html"/>
<item name="Distributed Cache Deploy" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html"/>
<item name="Support for YARN Shared Cache" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html"/>
+ <item name="Task Level Security Enforcement" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/TaskLevelSecurityEnforcement.html"/>
Review Comment:
Should it be TaskLevelSecurityEnforcement.**md**?
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/TaskLevelSecurityEnforcement.md
I could not find TaskLevelSecurityEnforcement.**html** in the source code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org