You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "K0K0V0K (via GitHub)" <gi...@apache.org> on 2026/03/05 10:30:27 UTC
[PR] YARN-11937. Yarn Proxy Behind a Reverse Proxy [hadoop]
K0K0V0K opened a new pull request, #8300:
URL: https://github.com/apache/hadoop/pull/8300
### Description of PR
When the Yarn Proxy is deployed behind a reverse proxy that is also used in application tracking URLs, the Yarn Proxy should redirect requests to that proxy instead of attempting to proxy them internally.
**Use Case**
Consider the following scenario:
• A user runs a Spark job.
• The Spark UI is hosted in the Spark History Server (SHS).
• Multiple SHS instances are deployed for high availability (HA).
• The tracking URL points to a Knox Gateway, which routes requests to the available SHS instances.
This setup ensures high availability for the tracking UI. If one SHS instance becomes unavailable, another can continue serving the UI.
**Problem Statement**
When the Knox Gateway forwards a user’s HTTP request to the Yarn Proxy, the Yarn Proxy attempts to proxy the request back to the Knox Gateway. However, this proxied request does not include the JWT token. As a result, Knox initiates authentication instead of forwarding the request to the appropriate SHS instance.
**Proposed Solution**
For security reasons, the JWT token must not be forwarded to the tracking URL. Therefore, when an application registers a tracking URL that includes a specific flag indicating that it is served behind a reverse proxy, the Yarn Proxy should redirect the user directly to the tracking URL instead of attempting to proxy the request internally.
Config
New config was created: yarn.web-proxy.redirect-flag
### How was this patch tested?
- UT was created
- Deployed a cluster with YARN, SPARK, KNOX and checked it there
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ ] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11937. Yarn Proxy Behind a Reverse Proxy [hadoop]
Posted by "roczei (via GitHub)" <gi...@apache.org>.
roczei commented on PR #8300:
URL: https://github.com/apache/hadoop/pull/8300#issuecomment-4091617186
Hi All,
Created a follow up PR for this because the current implementation might not correctly identify this parameter when it is part of a larger query string containing multiple parameters. For example in the following URL:
```
...?parameter1=true¶meter2=true&yarn_knox_proxy=true&doAs=user
```
Currently the logic expects the query string to be exactly yarn_knox_proxy=true, it will fail to recognize the flag when other parameters (like doAs, parameter1, or parameter2) are present.
The parameter detection should be improved to properly parse the query string and identify the yarn_knox_proxy=true key-value pair regardless of its position or the presence of other parameters.
Could you please take a look? Thank you!
https://github.com/apache/hadoop/pull/8357
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11937. Yarn Proxy Behind a Reverse Proxy [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8300:
URL: https://github.com/apache/hadoop/pull/8300#issuecomment-4004066969
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 20s | | https://github.com/apache/hadoop/pull/8300 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8300/1/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11937. Yarn Proxy Behind a Reverse Proxy [hadoop]
Posted by "p-szucs (via GitHub)" <gi...@apache.org>.
p-szucs commented on PR #8300:
URL: https://github.com/apache/hadoop/pull/8300#issuecomment-4010772235
Thanks for the patch @K0K0V0K, LGTM!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11937. Yarn Proxy Behind a Reverse Proxy [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 merged PR #8300:
URL: https://github.com/apache/hadoop/pull/8300
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11937. Yarn Proxy Behind a Reverse Proxy [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on PR #8300:
URL: https://github.com/apache/hadoop/pull/8300#issuecomment-4024509611
### Code review
Found 2 issues:
1. **NullPointerException when tracking URL has no query string** — `java.net.URI.getQuery()` returns `null` when the URI has no query component (e.g., `http://host:8188/history/application_001`). Calling `.equals(...)` on `null` will throw an NPE for any application whose tracking URL lacks a query string, which is the common case. The fix is to add a null guard: `toFetch.getQuery() != null && toFetch.getQuery().equals(...)` or flip the receiver to `(redirectFlagName + "=true").equals(toFetch.getQuery())`.
https://github.com/apache/hadoop/blob/3ee560a6bdf87786d178473ef66c78cf45358e34/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java#L553-L560
2. **Redirect is unreachable for FINISHED/KILLED/FAILED applications** — The new redirect-flag block is inserted after the application-state `switch`. For `KILLED`, `FINISHED`, and `FAILED` states, the switch already calls `ProxyUtils.sendRedirect(req, resp, toFetch.toString()); return;` and exits before reaching the new block. The new code only executes for `RUNNING` applications (the `default:` fall-through case). The PR's stated motivation is Knox Gateway with Spark History Server — which serves **completed (FINISHED) jobs** — so the feature will never trigger for its documented use case. The redirect-flag check needs to be placed before the state switch, or the switch cases need to be updated to respect the flag.
https://github.com/apache/hadoop/blob/3ee560a6bdf87786d178473ef66c78cf45358e34/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java#L527-L562
🤖 Generated with [Claude Code](https://claude.ai/code)
<sub>If this code review was useful, please react with 👍. Otherwise, react with 👎.</sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11937. Yarn Proxy Behind a Reverse Proxy [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8300:
URL: https://github.com/apache/hadoop/pull/8300#issuecomment-4029977254
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 24s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 56s | | trunk passed |
| +1 :green_heart: | compile | 4m 11s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | compile | 3m 51s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | checkstyle | 1m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 53s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | spotbugs | 3m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 51s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 3m 47s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javac | 3m 47s | | the patch passed |
| +1 :green_heart: | compile | 3m 47s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | javac | 3m 47s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 19s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8300/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 186 unchanged - 0 fixed = 187 total (was 186) |
| +1 :green_heart: | mvnsite | 1m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | spotbugs | 3m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 57s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 40s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 3m 54s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 1m 5s | | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 111m 19s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8300/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8300 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux f829e3db7be6 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3ee560a6bdf87786d178473ef66c78cf45358e34 |
| Default Java | Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8300/3/testReport/ |
| Max. process+thread count | 616 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8300/3/console |
| versions | git=2.43.0 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org