You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/20 21:45:34 UTC
[GitHub] [spark] xkrogen opened a new pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
xkrogen opened a new pull request #30450:
URL: https://github.com/apache/spark/pull/30450
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
This is a follow-on to PR #30096 which initially added support for printing direct links to the driver stdout/stderr logs from the application report output in `yarn.Client` using the `spark.yarn.includeDriverLogsLink` configuration. That PR made use of the ResourceManager's REST APIs to fetch the necessary information to construct the links. This PR proposes removing the dependency on the REST API, since the new logic is the only place in `yarn.Client` which makes use of this API, and instead leverages the RPC API via `YarnClient`, which brings the code in line with the rest of `yarn.Client`.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
While the old logic worked okay when running a Spark application in a "standard" environment with full access to Kerberos credentials, it can fail when run in an environment with restricted Kerberos credentials. In our case, this environment is represented by [Azkaban](https://azkaban.github.io/), but it likely affects other job scheduling systems as well. In such an environment, the application has delegation tokens which enabled it to communicate with services such as YARN, but the RM REST API is not typically covered by such delegation tokens (note that although YARN does actually support accessing the RM REST API via a delegation token as documented [here](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Delegation_Tokens_API), it is a new feature in alpha phase, and most deployments are likely not retrieving this token today).
Besides this enhancement, leveraging the `YarnClient` APIs greatly simplifies the processing logic, such as removing all JSON parsing.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Very minimal user-facing changes on top of PR #30096. Basically expands the scope of environments in which that feature will operate correctly.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
In addition to redoing the `spark-submit` testing as mentioned in PR #30096, I also tested this logic in a restricted-credentials environment (Azkaban). It succeeds where the previous logic would fail with a 401 error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731439552
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731423581
cc @mridulm @otterc @tgravescs @HeartSaVioR since you folks helped to review PR #30096
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-732797906
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-735900068
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731453464
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36054/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731461238
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36055/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30450:
URL: https://github.com/apache/spark/pull/30450#discussion_r528008087
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
##########
@@ -1192,32 +1187,31 @@ private[spark] class Client(
}
/**
- * Fetch links to the logs of the driver for the given application ID. This requires hitting the
- * RM REST API. Returns an empty map if the links could not be fetched. If this feature is
- * disabled via [[CLIENT_INCLUDE_DRIVER_LOGS_LINK]], an empty map is returned immediately.
+ * Fetch links to the logs of the driver for the given application report. This requires
+ * query the ResourceManager via RPC. Returns an empty map if the links could not be fetched.
+ * If this feature is disabled via [[CLIENT_INCLUDE_DRIVER_LOGS_LINK]], or if the application
+ * report indicates that the driver container isn't yet running
+ * (states `NEW`, `NEW_SAVING`, `SUBMITTED`, or `ACCEPTED`), an empty map is returned immediately.
*/
- private def getDriverLogsLink(appId: ApplicationId): IMap[String, String] = {
- if (!sparkConf.get(CLIENT_INCLUDE_DRIVER_LOGS_LINK)) {
+ private def getDriverLogsLink(appReport: ApplicationReport): IMap[String, String] = {
+ if (!sparkConf.get(CLIENT_INCLUDE_DRIVER_LOGS_LINK)
+ || appReport.getYarnApplicationState == YarnApplicationState.NEW
+ || appReport.getYarnApplicationState == YarnApplicationState.NEW_SAVING
+ || appReport.getYarnApplicationState == YarnApplicationState.SUBMITTED
+ || appReport.getYarnApplicationState == YarnApplicationState.ACCEPTED) {
return IMap()
}
try {
- val baseRmUrl = WebAppUtils.getRMWebAppURLWithScheme(hadoopConf)
- val response = ClientBuilder.newClient()
- .target(baseRmUrl)
- .path("ws").path("v1").path("cluster").path("apps")
- .path(appId.toString).path("appattempts")
- .request(MediaType.APPLICATION_JSON)
- .get()
- response.getStatusInfo.getFamily match {
- case Family.SUCCESSFUL => parseAppAttemptsJsonResponse(response.readEntity(classOf[String]))
- case _ =>
- logWarning(s"Unable to fetch app attempts info from $baseRmUrl, got "
- + s"status code ${response.getStatus}: ${response.getStatusInfo.getReasonPhrase}")
- IMap()
- }
+ val amContainerId = yarnClient
+ .getApplicationAttemptReport(appReport.getCurrentApplicationAttemptId)
+ .getAMContainerId
+ val baseUrl = yarnClient.getContainerReport(amContainerId).getLogUrl
Review comment:
Could the `baseUrl` be null. If yes, next line will throw NPE. Can we add a check for null. I do realize NPE is going to be handled by the catch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731467807
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731472429
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36057/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731467798
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36055/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-735905371
**[Test build #132002 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132002/testReport)** for PR 30450 at commit [`0bf9f5a`](https://github.com/apache/spark/commit/0bf9f5ac6b2685d1f24be5a2027d6daceceb2588).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731454678
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36052/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-732631463
**[Test build #131610 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131610/testReport)** for PR 30450 at commit [`93f3fe3`](https://github.com/apache/spark/commit/93f3fe3a8c94cd4297ab67f07ce0498c1628db74).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731427532
**[Test build #131446 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131446/testReport)** for PR 30450 at commit [`f160e78`](https://github.com/apache/spark/commit/f160e7865bd9b9a018be29700116781610252946).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731472443
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-735960525
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-732303149
re: unit testing, @mmuralid made me aware of `BaseYarnClusterSuite` which is spinning up a MiniYARN cluster. I'll try to put together a test leveraging this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-736033003
Thanks @mridulm @otterc @HeartSaVioR ! Much appreciated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-735960525
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-732585240
**[Test build #131610 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131610/testReport)** for PR 30450 at commit [`93f3fe3`](https://github.com/apache/spark/commit/93f3fe3a8c94cd4297ab67f07ce0498c1628db74).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #30450:
URL: https://github.com/apache/spark/pull/30450#discussion_r528009005
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
##########
@@ -1192,32 +1192,31 @@ private[spark] class Client(
}
/**
- * Fetch links to the logs of the driver for the given application ID. This requires hitting the
- * RM REST API. Returns an empty map if the links could not be fetched. If this feature is
- * disabled via [[CLIENT_INCLUDE_DRIVER_LOGS_LINK]], an empty map is returned immediately.
+ * Fetch links to the logs of the driver for the given application report. This requires
+ * query the ResourceManager via RPC. Returns an empty map if the links could not be fetched.
+ * If this feature is disabled via [[CLIENT_INCLUDE_DRIVER_LOGS_LINK]], or if the application
+ * report indicates that the driver container isn't yet running
+ * (states `NEW`, `NEW_SAVING`, `SUBMITTED`, or `ACCEPTED`), an empty map is returned immediately.
*/
- private def getDriverLogsLink(appId: ApplicationId): IMap[String, String] = {
- if (!sparkConf.get(CLIENT_INCLUDE_DRIVER_LOGS_LINK)) {
+ private def getDriverLogsLink(appReport: ApplicationReport): IMap[String, String] = {
+ if (!sparkConf.get(CLIENT_INCLUDE_DRIVER_LOGS_LINK)
+ || appReport.getYarnApplicationState == YarnApplicationState.NEW
+ || appReport.getYarnApplicationState == YarnApplicationState.NEW_SAVING
+ || appReport.getYarnApplicationState == YarnApplicationState.SUBMITTED
+ || appReport.getYarnApplicationState == YarnApplicationState.ACCEPTED) {
return IMap()
}
try {
- val baseRmUrl = WebAppUtils.getRMWebAppURLWithScheme(hadoopConf)
- val response = ClientBuilder.newClient()
- .target(baseRmUrl)
- .path("ws").path("v1").path("cluster").path("apps")
- .path(appId.toString).path("appattempts")
- .request(MediaType.APPLICATION_JSON)
- .get()
- response.getStatusInfo.getFamily match {
- case Family.SUCCESSFUL => parseAppAttemptsJsonResponse(response.readEntity(classOf[String]))
- case _ =>
- logWarning(s"Unable to fetch app attempts info from $baseRmUrl, got "
- + s"status code ${response.getStatus}: ${response.getStatusInfo.getReasonPhrase}")
- IMap()
- }
+ val amContainerId = yarnClient
+ .getApplicationAttemptReport(appReport.getCurrentApplicationAttemptId)
Review comment:
Arg. Added an `.editorconfig`, hopefully won't have this problem anymore. Thank you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-732638291
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #30450:
URL: https://github.com/apache/spark/pull/30450#discussion_r528010913
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
##########
@@ -1192,32 +1187,31 @@ private[spark] class Client(
}
/**
- * Fetch links to the logs of the driver for the given application ID. This requires hitting the
- * RM REST API. Returns an empty map if the links could not be fetched. If this feature is
- * disabled via [[CLIENT_INCLUDE_DRIVER_LOGS_LINK]], an empty map is returned immediately.
+ * Fetch links to the logs of the driver for the given application report. This requires
+ * query the ResourceManager via RPC. Returns an empty map if the links could not be fetched.
+ * If this feature is disabled via [[CLIENT_INCLUDE_DRIVER_LOGS_LINK]], or if the application
+ * report indicates that the driver container isn't yet running
+ * (states `NEW`, `NEW_SAVING`, `SUBMITTED`, or `ACCEPTED`), an empty map is returned immediately.
*/
- private def getDriverLogsLink(appId: ApplicationId): IMap[String, String] = {
- if (!sparkConf.get(CLIENT_INCLUDE_DRIVER_LOGS_LINK)) {
+ private def getDriverLogsLink(appReport: ApplicationReport): IMap[String, String] = {
+ if (!sparkConf.get(CLIENT_INCLUDE_DRIVER_LOGS_LINK)
+ || appReport.getYarnApplicationState == YarnApplicationState.NEW
+ || appReport.getYarnApplicationState == YarnApplicationState.NEW_SAVING
+ || appReport.getYarnApplicationState == YarnApplicationState.SUBMITTED
+ || appReport.getYarnApplicationState == YarnApplicationState.ACCEPTED) {
return IMap()
}
try {
- val baseRmUrl = WebAppUtils.getRMWebAppURLWithScheme(hadoopConf)
- val response = ClientBuilder.newClient()
- .target(baseRmUrl)
- .path("ws").path("v1").path("cluster").path("apps")
- .path(appId.toString).path("appattempts")
- .request(MediaType.APPLICATION_JSON)
- .get()
- response.getStatusInfo.getFamily match {
- case Family.SUCCESSFUL => parseAppAttemptsJsonResponse(response.readEntity(classOf[String]))
- case _ =>
- logWarning(s"Unable to fetch app attempts info from $baseRmUrl, got "
- + s"status code ${response.getStatus}: ${response.getStatusInfo.getReasonPhrase}")
- IMap()
- }
+ val amContainerId = yarnClient
+ .getApplicationAttemptReport(appReport.getCurrentApplicationAttemptId)
+ .getAMContainerId
+ val baseUrl = yarnClient.getContainerReport(amContainerId).getLogUrl
Review comment:
Yeah, it seems like all of these calls can return null. I updated this block to be much more defensive.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731426510
**[Test build #131446 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131446/testReport)** for PR 30450 at commit [`f160e78`](https://github.com/apache/spark/commit/f160e7865bd9b9a018be29700116781610252946).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731439560
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131448/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731447832
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731467810
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36055/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731427554
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131446/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-735897388
**[Test build #132001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132001/testReport)** for PR 30450 at commit [`3f9013d`](https://github.com/apache/spark/commit/3f9013d8acd4a1d4cdb82d00bbcce41862988759).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731427546
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #30450:
URL: https://github.com/apache/spark/pull/30450#discussion_r532739914
##########
File path: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala
##########
@@ -230,6 +230,37 @@ class YarnClusterSuite extends BaseYarnClusterSuite {
}
}
+ test("running Spark in yarn-cluster mode displays driver log links") {
+ val log4jConf = new File(tempDir, "log4j.properties")
+ val logOutFile = new File(tempDir, "logs")
+ Files.write(
+ s"""log4j.rootCategory=DEBUG,file
+ |log4j.appender.file=org.apache.log4j.FileAppender
+ |log4j.appender.file.file=$logOutFile
+ |log4j.appender.file.layout=org.apache.log4j.PatternLayout
+ |""".stripMargin,
+ log4jConf, StandardCharsets.UTF_8)
+ // Since this test is trying to extract log output from the SparkSubmit process itself,
+ // standard options to the Spark process don't take effect. Leverage the java-opts file which
+ // will get picked up for the SparkSubmit process.
+ val confDir = new File(tempDir, "conf")
+ confDir.mkdir()
+ val javaOptsFile = new File(confDir, "java-opts")
+ Files.write(s"-Dlog4j.configuration=file://$log4jConf\n", javaOptsFile, StandardCharsets.UTF_8)
+
+ val result = File.createTempFile("result", null, tempDir)
+ val finalState = runSpark(clientMode = false,
+ mainClassName(YarnClusterDriver.getClass),
+ appArgs = Seq(result.getAbsolutePath),
+ extraEnv = Map("SPARK_CONF_DIR" -> confDir.getAbsolutePath),
+ extraConf = Map("spark.yarn.includeDriverLogsLink" -> true.toString))
+ checkResult(finalState, result)
+ val logOutput = Files.toString(logOutFile, StandardCharsets.UTF_8)
+ val logFilePattern = raw"""(?s).+\sDriver Logs \(<NAME>\): http://.+/<NAME>\?start=-4096\s.+"""
Review comment:
Updated to accept any query parameters, and also accept either http or https.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731435312
**[Test build #131448 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131448/testReport)** for PR 30450 at commit [`cd27c14`](https://github.com/apache/spark/commit/cd27c144686426cebe1850476a647d75a1c6d4fa).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731426510
**[Test build #131446 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131446/testReport)** for PR 30450 at commit [`f160e78`](https://github.com/apache/spark/commit/f160e7865bd9b9a018be29700116781610252946).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-733500240
Given vacation time, I will leave this open so that @otterc @tgravescs and @HeartSaVioR have sufficient time to get to it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-735897388
**[Test build #132001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132001/testReport)** for PR 30450 at commit [`3f9013d`](https://github.com/apache/spark/commit/3f9013d8acd4a1d4cdb82d00bbcce41862988759).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731440760
**[Test build #131449 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131449/testReport)** for PR 30450 at commit [`c966575`](https://github.com/apache/spark/commit/c966575ea690b78265c466341ac34b17453e1c96).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731447988
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731455106
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-732797906
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #30450:
URL: https://github.com/apache/spark/pull/30450#discussion_r532287534
##########
File path: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala
##########
@@ -230,6 +230,37 @@ class YarnClusterSuite extends BaseYarnClusterSuite {
}
}
+ test("running Spark in yarn-cluster mode displays driver log links") {
+ val log4jConf = new File(tempDir, "log4j.properties")
+ val logOutFile = new File(tempDir, "logs")
+ Files.write(
+ s"""log4j.rootCategory=DEBUG,file
+ |log4j.appender.file=org.apache.log4j.FileAppender
+ |log4j.appender.file.file=$logOutFile
+ |log4j.appender.file.layout=org.apache.log4j.PatternLayout
+ |""".stripMargin,
+ log4jConf, StandardCharsets.UTF_8)
+ // Since this test is trying to extract log output from the SparkSubmit process itself,
+ // standard options to the Spark process don't take effect. Leverage the java-opts file which
+ // will get picked up for the SparkSubmit process.
+ val confDir = new File(tempDir, "conf")
+ confDir.mkdir()
+ val javaOptsFile = new File(confDir, "java-opts")
+ Files.write(s"-Dlog4j.configuration=file://$log4jConf\n", javaOptsFile, StandardCharsets.UTF_8)
+
+ val result = File.createTempFile("result", null, tempDir)
+ val finalState = runSpark(clientMode = false,
+ mainClassName(YarnClusterDriver.getClass),
+ appArgs = Seq(result.getAbsolutePath),
+ extraEnv = Map("SPARK_CONF_DIR" -> confDir.getAbsolutePath),
+ extraConf = Map("spark.yarn.includeDriverLogsLink" -> true.toString))
+ checkResult(finalState, result)
+ val logOutput = Files.toString(logOutFile, StandardCharsets.UTF_8)
+ val logFilePattern = raw"""(?s).+\sDriver Logs \(<NAME>\): http://.+/<NAME>\?start=-4096\s.+"""
Review comment:
If we no longer generate the url, it's safer to exclude the part (like query parameters) on validation which could be easily changed.
What we are sure for now is only that `<NAME>` is placed for the last part of path, and that said, the URL should either end, or the remaining part should start with `?`.
That's a nit anyway, so OK to merge without making change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731463945
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731466865
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36057/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #30450:
URL: https://github.com/apache/spark/pull/30450#discussion_r532730280
##########
File path: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala
##########
@@ -230,6 +230,37 @@ class YarnClusterSuite extends BaseYarnClusterSuite {
}
}
+ test("running Spark in yarn-cluster mode displays driver log links") {
+ val log4jConf = new File(tempDir, "log4j.properties")
+ val logOutFile = new File(tempDir, "logs")
+ Files.write(
+ s"""log4j.rootCategory=DEBUG,file
+ |log4j.appender.file=org.apache.log4j.FileAppender
+ |log4j.appender.file.file=$logOutFile
+ |log4j.appender.file.layout=org.apache.log4j.PatternLayout
+ |""".stripMargin,
+ log4jConf, StandardCharsets.UTF_8)
+ // Since this test is trying to extract log output from the SparkSubmit process itself,
+ // standard options to the Spark process don't take effect. Leverage the java-opts file which
+ // will get picked up for the SparkSubmit process.
+ val confDir = new File(tempDir, "conf")
+ confDir.mkdir()
+ val javaOptsFile = new File(confDir, "java-opts")
+ Files.write(s"-Dlog4j.configuration=file://$log4jConf\n", javaOptsFile, StandardCharsets.UTF_8)
+
+ val result = File.createTempFile("result", null, tempDir)
+ val finalState = runSpark(clientMode = false,
+ mainClassName(YarnClusterDriver.getClass),
+ appArgs = Seq(result.getAbsolutePath),
+ extraEnv = Map("SPARK_CONF_DIR" -> confDir.getAbsolutePath),
+ extraConf = Map("spark.yarn.includeDriverLogsLink" -> true.toString))
+ checkResult(finalState, result)
+ val logOutput = Files.toString(logOutFile, StandardCharsets.UTF_8)
+ val logFilePattern = raw"""(?s).+\sDriver Logs \(<NAME>\): http://.+/<NAME>\?start=-4096\s.+"""
Review comment:
Great call, yes. I have fixed this to match any number for the start parameter, and match whether or not `?start=NNNN` is present. I just re-read your comment @HeartSaVioR and maybe it does make more sense to ignore the parameters altogether. Will update again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30450:
URL: https://github.com/apache/spark/pull/30450#discussion_r528002512
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
##########
@@ -1192,32 +1192,31 @@ private[spark] class Client(
}
/**
- * Fetch links to the logs of the driver for the given application ID. This requires hitting the
- * RM REST API. Returns an empty map if the links could not be fetched. If this feature is
- * disabled via [[CLIENT_INCLUDE_DRIVER_LOGS_LINK]], an empty map is returned immediately.
+ * Fetch links to the logs of the driver for the given application report. This requires
+ * query the ResourceManager via RPC. Returns an empty map if the links could not be fetched.
+ * If this feature is disabled via [[CLIENT_INCLUDE_DRIVER_LOGS_LINK]], or if the application
+ * report indicates that the driver container isn't yet running
+ * (states `NEW`, `NEW_SAVING`, `SUBMITTED`, or `ACCEPTED`), an empty map is returned immediately.
*/
- private def getDriverLogsLink(appId: ApplicationId): IMap[String, String] = {
- if (!sparkConf.get(CLIENT_INCLUDE_DRIVER_LOGS_LINK)) {
+ private def getDriverLogsLink(appReport: ApplicationReport): IMap[String, String] = {
+ if (!sparkConf.get(CLIENT_INCLUDE_DRIVER_LOGS_LINK)
+ || appReport.getYarnApplicationState == YarnApplicationState.NEW
+ || appReport.getYarnApplicationState == YarnApplicationState.NEW_SAVING
+ || appReport.getYarnApplicationState == YarnApplicationState.SUBMITTED
+ || appReport.getYarnApplicationState == YarnApplicationState.ACCEPTED) {
return IMap()
}
try {
- val baseRmUrl = WebAppUtils.getRMWebAppURLWithScheme(hadoopConf)
- val response = ClientBuilder.newClient()
- .target(baseRmUrl)
- .path("ws").path("v1").path("cluster").path("apps")
- .path(appId.toString).path("appattempts")
- .request(MediaType.APPLICATION_JSON)
- .get()
- response.getStatusInfo.getFamily match {
- case Family.SUCCESSFUL => parseAppAttemptsJsonResponse(response.readEntity(classOf[String]))
- case _ =>
- logWarning(s"Unable to fetch app attempts info from $baseRmUrl, got "
- + s"status code ${response.getStatus}: ${response.getStatusInfo.getReasonPhrase}")
- IMap()
- }
+ val amContainerId = yarnClient
+ .getApplicationAttemptReport(appReport.getCurrentApplicationAttemptId)
Review comment:
Nit: indentation should be 2
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
##########
@@ -1192,32 +1192,31 @@ private[spark] class Client(
}
/**
- * Fetch links to the logs of the driver for the given application ID. This requires hitting the
- * RM REST API. Returns an empty map if the links could not be fetched. If this feature is
- * disabled via [[CLIENT_INCLUDE_DRIVER_LOGS_LINK]], an empty map is returned immediately.
+ * Fetch links to the logs of the driver for the given application report. This requires
+ * query the ResourceManager via RPC. Returns an empty map if the links could not be fetched.
+ * If this feature is disabled via [[CLIENT_INCLUDE_DRIVER_LOGS_LINK]], or if the application
+ * report indicates that the driver container isn't yet running
+ * (states `NEW`, `NEW_SAVING`, `SUBMITTED`, or `ACCEPTED`), an empty map is returned immediately.
*/
- private def getDriverLogsLink(appId: ApplicationId): IMap[String, String] = {
- if (!sparkConf.get(CLIENT_INCLUDE_DRIVER_LOGS_LINK)) {
+ private def getDriverLogsLink(appReport: ApplicationReport): IMap[String, String] = {
+ if (!sparkConf.get(CLIENT_INCLUDE_DRIVER_LOGS_LINK)
+ || appReport.getYarnApplicationState == YarnApplicationState.NEW
Review comment:
Nit: indentation should be 2
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731472443
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-735900028
**[Test build #132001 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132001/testReport)** for PR 30450 at commit [`3f9013d`](https://github.com/apache/spark/commit/3f9013d8acd4a1d4cdb82d00bbcce41862988759).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731439552
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-735901016
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731435312
**[Test build #131448 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131448/testReport)** for PR 30450 at commit [`cd27c14`](https://github.com/apache/spark/commit/cd27c144686426cebe1850476a647d75a1c6d4fa).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731467807
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731446529
@otterc regarding a unit test, I did think about this but it looks like to unit test it, we'll need to do some refactoring to pull `YarnClient` out, and mock its responses. It should be doable, just not sure if the setup effort is worth it for this small patch. FWIW the previous unit tests were only testing the parsing logic, which is now handled by YARN code.
I'm happy to put in the refactoring work for a unit test if others feel it's necessary here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731455106
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-736028441
Given the approvals, I am merging this in. Thanks for working on this @xkrogen !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731440760
**[Test build #131449 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131449/testReport)** for PR 30450 at commit [`c966575`](https://github.com/apache/spark/commit/c966575ea690b78265c466341ac34b17453e1c96).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-732638291
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731463945
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731454693
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731463953
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36054/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731463930
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36054/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-735905371
**[Test build #132002 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132002/testReport)** for PR 30450 at commit [`0bf9f5a`](https://github.com/apache/spark/commit/0bf9f5ac6b2685d1f24be5a2027d6daceceb2588).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #30450:
URL: https://github.com/apache/spark/pull/30450#discussion_r530136636
##########
File path: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala
##########
@@ -230,6 +230,37 @@ class YarnClusterSuite extends BaseYarnClusterSuite {
}
}
+ test("running Spark in yarn-cluster mode displays driver log links") {
+ val log4jConf = new File(tempDir, "log4j.properties")
+ val logOutFile = new File(tempDir, "logs")
+ Files.write(
+ s"""log4j.rootCategory=DEBUG,file
+ |log4j.appender.file=org.apache.log4j.FileAppender
+ |log4j.appender.file.file=$logOutFile
+ |log4j.appender.file.layout=org.apache.log4j.PatternLayout
+ |""".stripMargin,
+ log4jConf, StandardCharsets.UTF_8)
+ // Since this test is trying to extract log output from the SparkSubmit process itself,
+ // standard options to the Spark process don't take effect. Leverage the java-opts file which
+ // will get picked up for the SparkSubmit process.
+ val confDir = new File(tempDir, "conf")
+ confDir.mkdir()
+ val javaOptsFile = new File(confDir, "java-opts")
+ Files.write(s"-Dlog4j.configuration=file://$log4jConf\n", javaOptsFile, StandardCharsets.UTF_8)
+
+ val result = File.createTempFile("result", null, tempDir)
+ val finalState = runSpark(clientMode = false,
+ mainClassName(YarnClusterDriver.getClass),
+ appArgs = Seq(result.getAbsolutePath),
+ extraEnv = Map("SPARK_CONF_DIR" -> confDir.getAbsolutePath),
+ extraConf = Map("spark.yarn.includeDriverLogsLink" -> true.toString))
+ checkResult(finalState, result)
+ val logOutput = Files.toString(logOutFile, StandardCharsets.UTF_8)
+ val logFilePattern = raw"""(?s).+\sDriver Logs \(<NAME>\): http://.+/<NAME>\?start=-4096\s.+"""
Review comment:
nit: replace 4096 with regex for number ? Since we are not generating the url anymore.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731447939
**[Test build #131451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131451/testReport)** for PR 30450 at commit [`7c00831`](https://github.com/apache/spark/commit/7c00831eb2f41ca856ff78b09df1b55b5e1b4164).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] asfgit closed pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #30450:
URL: https://github.com/apache/spark/pull/30450
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-735901016
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731446771
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36052/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-735927480
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731455001
**[Test build #131451 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131451/testReport)** for PR 30450 at commit [`7c00831`](https://github.com/apache/spark/commit/7c00831eb2f41ca856ff78b09df1b55b5e1b4164).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731447988
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-735927480
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-735900068
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-735917356
**[Test build #132002 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132002/testReport)** for PR 30450 at commit [`0bf9f5a`](https://github.com/apache/spark/commit/0bf9f5ac6b2685d1f24be5a2027d6daceceb2588).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-732585240
**[Test build #131610 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131610/testReport)** for PR 30450 at commit [`93f3fe3`](https://github.com/apache/spark/commit/93f3fe3a8c94cd4297ab67f07ce0498c1628db74).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731454693
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-732509456
Managed to get a test going in `YarnClusterSuite`. It was more challenging than expected because capturing the output of `yarn.Client`, which runs within the `spark-submit` process, isn't very straightforward, since Spark doesn't provide many mechanisms to customize that environment (understandably).
This led me to discover a bug that the report fetch can fail after the app stops running, presumably since the driver container no longer exists. I limited the driver log fetch to only `RUNNING` state apps.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731439515
**[Test build #131448 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131448/testReport)** for PR 30450 at commit [`cd27c14`](https://github.com/apache/spark/commit/cd27c144686426cebe1850476a647d75a1c6d4fa).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30450:
URL: https://github.com/apache/spark/pull/30450#issuecomment-731427546
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org