You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/02/25 06:50:59 UTC
[GitHub] [spark] sunchao opened a new pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
sunchao opened a new pull request #31642:
URL: https://github.com/apache/spark/pull/31642
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This adds `hadoop-yarn-server-web-proxy` as dependency for Yarn and Hadoop 3.x profile (it is already a dependency for 2.x). Also excludes some dependencies from the module which are already covered by other Hadoop jars used by Spark.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
The class `org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter` is used by `ApplicationMaster`:
```scala
private def addAmIpFilter(driver: Option[RpcEndpointRef], proxyBase: String) = {
val amFilter = "org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter"
val params = client.getAmIpFilterParams(yarnConf, proxyBase)
driver match {
case Some(d) =>
d.send(AddWebUIFilter(amFilter, params, proxyBase))
...
```
and will be loaded at runtime. Therefore, without the above jar Spark Yarn app will fail with `ClassNotFoundError`.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Existing unit tests. Also tested manually and it worked with the fix, while was failing previously.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786403501
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135483/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-785706873
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40039/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-787003431
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135525/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #31642: [SPARK-33212][FOLLOWUP] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #31642:
URL: https://github.com/apache/spark/pull/31642
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-787002690
**[Test build #135525 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135525/testReport)** for PR 31642 at commit [`e2d32aa`](https://github.com/apache/spark/commit/e2d32aa88ef6db801726d6385388696ce2fcac7f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786331963
**[Test build #135483 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135483/testReport)** for PR 31642 at commit [`17a63bd`](https://github.com/apache/spark/commit/17a63bdcee5503c57e20f4e2d23b20cc3ca14931).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-787036789
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135532/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-787015036
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40113/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-787560097
Thank you for the summary. For those test cases, I'll take a look in this week, @sunchao .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-787017537
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40113/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786421443
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135485/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786805816
**[Test build #135519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135519/testReport)** for PR 31642 at commit [`1de8590`](https://github.com/apache/spark/commit/1de859042bfbb7056cd1276aaf97686254f0c2ce).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-787036789
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135532/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786416280
**[Test build #135485 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135485/testReport)** for PR 31642 at commit [`e2d32aa`](https://github.com/apache/spark/commit/e2d32aa88ef6db801726d6385388696ce2fcac7f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786331963
**[Test build #135483 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135483/testReport)** for PR 31642 at commit [`17a63bd`](https://github.com/apache/spark/commit/17a63bdcee5503c57e20f4e2d23b20cc3ca14931).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786850585
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40100/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786355097
**[Test build #135485 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135485/testReport)** for PR 31642 at commit [`e2d32aa`](https://github.com/apache/spark/commit/e2d32aa88ef6db801726d6385388696ce2fcac7f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-787030191
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40113/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786035316
This seems like a decent workaround, but is it the right long-term fix? It seems like depending on the non-shaded JAR breaks the benefits of leveraging only the shaded `hadoop-client-api` and `hadoop-client-runtime` JARs.
Should `AmIpFilter` be a public class, i.e., is it something AMs are expected to leverage? If yes -- can we get it moved to `hadoop-client-runtime`? If no -- should we stop using it here and re-implement some of the required logic ourselves?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-787030191
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40113/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786386587
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40065/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786366418
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40065/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786905402
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135519/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786421443
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135485/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786817839
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40100/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786257114
This is a small jar though and we've striped all its transitive dependencies. Agree that in the long term we should either: 1) consider to re-implement the logic in Spark which allows us to get away from server-side dependency in Hadoop. cc @tgravescs for more insights on this. 2) move the class to `hadoop-client-runtime` as you suggested.
I need to take a look at the test failures - seems `WebAppProxyServlet` was pulled from `hadoop-yarn-server-web-proxy` jar instead of `hadoop-client-minicluster`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786989063
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40106/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786386587
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40065/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786383437
I checked the Jenkins log. It's testing `Hadoop 3.2` correctly. I'll fix the PR title.
```
[info] Building Spark using Maven with these arguments: -Phadoop-3.2 -Phive-2.3 -Pyarn -Phadoop-cloud -Pkinesis-asl -Pkubernetes -Phive-thriftserver -Phive -Pspark-ganglia-lgpl -Pmesos clean package -DskipTests
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786367088
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40063/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-787400523
@dongjoon-hyun Github CI all succeeded (except the flaky tests).
For the Jenkins CI, here are the results:
- [3.2](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135485/)
- [2.7](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135525/)
The tests all passed except two in SQL:
```
Caused by: java.lang.IllegalArgumentException: Unknown key pii
at org.apache.orc.impl.HadoopShimsPre2_3$NullKeyProvider.getCurrentKeyVersion(HadoopShimsPre2_3.java:71)
at org.apache.orc.impl.WriterImpl.getKey(WriterImpl.java:871)
at org.apache.orc.impl.WriterImpl.visitTypeTree(WriterImpl.java:902)
at org.apache.orc.impl.WriterImpl.visitTypeTree(WriterImpl.java:915)
at org.apache.orc.impl.WriterImpl.setupEncryption(WriterImpl.java:938)
at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:162)
at org.apache.orc.OrcFile.createWriter(OrcFile.java:1004)
at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:47)
at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:120)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:126)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:111)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:270)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$15(FileFormatWriter.scala:210)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:498)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1437)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
- Write and read an encrypted file *** FAILED ***
org.apache.spark.SparkException: Job aborted.
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:232)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:188)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:108)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:106)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:131)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
```
which doesn't look like related as this PR only touches dependencies for YARN.
Also the above succeeded in Jenkins CI if I remove `[test-maven]`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-785714299
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40039/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786382726
@sunchao . The PR title looks misleading. Which hadoop profile are you testing now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #31642:
URL: https://github.com/apache/spark/pull/31642#discussion_r583316329
##########
File path: assembly/pom.xml
##########
@@ -136,6 +136,10 @@
<artifactId>spark-yarn_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
Review comment:
This artifact is only needed at runtime so included here in distribution. Ideally we should declare it as a runtime dependency. However, the `WebAppProxyServlet` class in the jar conflicts with the same one from `hadoop-cluster-client-minicluster` during tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-787003431
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135525/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786902711
**[Test build #135519 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135519/testReport)** for PR 31642 at commit [`1de8590`](https://github.com/apache/spark/commit/1de859042bfbb7056cd1276aaf97686254f0c2ce).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-787116531
Could you confirm which links are the two successful runs (Hadoop 3.2/2.7)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786355097
**[Test build #135485 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135485/testReport)** for PR 31642 at commit [`e2d32aa`](https://github.com/apache/spark/commit/e2d32aa88ef6db801726d6385388696ce2fcac7f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-785694992
**[Test build #135459 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135459/testReport)** for PR 31642 at commit [`17a63bd`](https://github.com/apache/spark/commit/17a63bdcee5503c57e20f4e2d23b20cc3ca14931).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786905402
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135519/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-785714299
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40039/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786310603
Thank you for the follow-up, @sunchao !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786405169
It's not possible for the single title because one Jenkins run only will pick-up a single profile.
What you need is to run two Jenkins runs. So, you need to trigger twice like the following.
1. Set the title with `test-hadoop3.2` and trigger.
2. Wait 5 or 10 minutes until Jenkins posts a comment. Then, check the log.
3. Then, rename the title with test-hadoop3.2 and trigger.
4. Wait another 5 or 10 minutes until Jenkins posts a comment. Then, check the log.
Then, you will have effectively concurrent two runs (for both profiles)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786833307
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40100/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-787012019
**[Test build #135532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135532/testReport)** for PR 31642 at commit [`c31e06d`](https://github.com/apache/spark/commit/c31e06ddebb7ac5c6b1f63f72c8a82fb060c05ce).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-785761042
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135459/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786429739
Thanks @dongjoon-hyun . Let me trigger another run for Hadoop 2.7 just to be sure.
Jenkins CI says there're 2 failures in SQL module but I'm guessing they're not related (this PR doesn't touch anything in SQL), although I can't seem to find which two tests are these.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-787034162
**[Test build #135532 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135532/testReport)** for PR 31642 at commit [`c31e06d`](https://github.com/apache/spark/commit/c31e06ddebb7ac5c6b1f63f72c8a82fb060c05ce).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-785709141
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40039/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-785761042
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135459/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786397168
**[Test build #135483 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135483/testReport)** for PR 31642 at commit [`17a63bd`](https://github.com/apache/spark/commit/17a63bdcee5503c57e20f4e2d23b20cc3ca14931).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-785760980
**[Test build #135459 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135459/testReport)** for PR 31642 at commit [`17a63bd`](https://github.com/apache/spark/commit/17a63bdcee5503c57e20f4e2d23b20cc3ca14931).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786971184
**[Test build #135525 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135525/testReport)** for PR 31642 at commit [`e2d32aa`](https://github.com/apache/spark/commit/e2d32aa88ef6db801726d6385388696ce2fcac7f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786971184
**[Test build #135525 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135525/testReport)** for PR 31642 at commit [`e2d32aa`](https://github.com/apache/spark/commit/e2d32aa88ef6db801726d6385388696ce2fcac7f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-787012019
**[Test build #135532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135532/testReport)** for PR 31642 at commit [`c31e06d`](https://github.com/apache/spark/commit/c31e06ddebb7ac5c6b1f63f72c8a82fb060c05ce).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786805816
**[Test build #135519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135519/testReport)** for PR 31642 at commit [`1de8590`](https://github.com/apache/spark/commit/1de859042bfbb7056cd1276aaf97686254f0c2ce).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786850585
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40100/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786375345
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40065/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786310408
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-785694992
**[Test build #135459 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135459/testReport)** for PR 31642 at commit [`17a63bd`](https://github.com/apache/spark/commit/17a63bdcee5503c57e20f4e2d23b20cc3ca14931).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786989063
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40106/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786367088
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40063/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786403501
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135483/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786385637
Sorry I wanted to test 2.7 and 3.2 at the same time. Is that possible?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #31642: [SPARK-33212][FOLLOWUP] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-788091996
yes AmIpFilter is expected to be used by ApplicationMasters so ideally Hadoop would move it to somewhere easily accessible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786359456
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40063/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31642: [SPARK-33212][FOLLOWUP][test-maven][test-hadoop3.2][test-hadoop2.7] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31642:
URL: https://github.com/apache/spark/pull/31642#issuecomment-786349301
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40063/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org