You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2017/02/10 13:44:33 UTC
spark git commit: [SPARK-19545][YARN] Fix compile issue for Spark on
Yarn when building against Hadoop 2.6.0~2.6.3
Repository: spark
Updated Branches:
refs/heads/master d5593f7f5 -> 8e8afb3a3
[SPARK-19545][YARN] Fix compile issue for Spark on Yarn when building against Hadoop 2.6.0~2.6.3
## What changes were proposed in this pull request?
Due to the newly added API in Hadoop 2.6.4+, Spark builds against Hadoop 2.6.0~2.6.3 will meet compile error. So here still reverting back to use reflection to handle this issue.
## How was this patch tested?
Manual verification.
Author: jerryshao <ss...@hortonworks.com>
Closes #16884 from jerryshao/SPARK-19545.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8e8afb3a
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8e8afb3a
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8e8afb3a
Branch: refs/heads/master
Commit: 8e8afb3a3468aa743d13e23e10e77e94b772b2ed
Parents: d5593f7
Author: jerryshao <ss...@hortonworks.com>
Authored: Fri Feb 10 13:44:26 2017 +0000
Committer: Sean Owen <so...@cloudera.com>
Committed: Fri Feb 10 13:44:26 2017 +0000
----------------------------------------------------------------------
docs/running-on-yarn.md | 6 ++---
.../org/apache/spark/deploy/yarn/Client.scala | 25 ++++++++++++++++----
2 files changed, 23 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/8e8afb3a/docs/running-on-yarn.md
----------------------------------------------------------------------
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md
index c95f627..cf95b95 100644
--- a/docs/running-on-yarn.md
+++ b/docs/running-on-yarn.md
@@ -445,7 +445,7 @@ To use a custom metrics.properties for the application master and executors, upd
This will be used with YARN's rolling log aggregation, to enable this feature in YARN side
<code>yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds</code> should be
configured in yarn-site.xml.
- This feature can only be used with Hadoop 2.6.1+. The Spark log4j appender needs be changed to use
+ This feature can only be used with Hadoop 2.6.4+. The Spark log4j appender needs be changed to use
FileAppender or another appender that can handle the files being removed while its running. Based
on the file name configured in the log4j configuration (like spark.log), the user should set the
regex (spark*) to include all the log files that need to be aggregated.
@@ -524,8 +524,8 @@ pre-packaged distribution.
1. In the `yarn-site.xml` on each node, add `spark_shuffle` to `yarn.nodemanager.aux-services`,
then set `yarn.nodemanager.aux-services.spark_shuffle.class` to
`org.apache.spark.network.yarn.YarnShuffleService`.
-1. Increase `NodeManager's` heap size by setting `YARN_HEAPSIZE` (1000 by default) in `etc/hadoop/yarn-env.sh`
-to avoid garbage collection issues during shuffle.
+1. Increase `NodeManager's` heap size by setting `YARN_HEAPSIZE` (1000 by default) in `etc/hadoop/yarn-env.sh`
+to avoid garbage collection issues during shuffle.
1. Restart all `NodeManager`s in your cluster.
The following extra configuration options are available when the shuffle service is running on YARN:
http://git-wip-us.apache.org/repos/asf/spark/blob/8e8afb3a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
----------------------------------------------------------------------
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
index 635c1ac..70826ed 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
@@ -245,12 +245,27 @@ private[spark] class Client(
}
sparkConf.get(ROLLED_LOG_INCLUDE_PATTERN).foreach { includePattern =>
- val logAggregationContext = Records.newRecord(classOf[LogAggregationContext])
- logAggregationContext.setRolledLogsIncludePattern(includePattern)
- sparkConf.get(ROLLED_LOG_EXCLUDE_PATTERN).foreach { excludePattern =>
- logAggregationContext.setRolledLogsExcludePattern(excludePattern)
+ try {
+ val logAggregationContext = Records.newRecord(classOf[LogAggregationContext])
+
+ // These two methods were added in Hadoop 2.6.4, so we still need to use reflection to
+ // avoid compile error when building against Hadoop 2.6.0 ~ 2.6.3.
+ val setRolledLogsIncludePatternMethod =
+ logAggregationContext.getClass.getMethod("setRolledLogsIncludePattern", classOf[String])
+ setRolledLogsIncludePatternMethod.invoke(logAggregationContext, includePattern)
+
+ sparkConf.get(ROLLED_LOG_EXCLUDE_PATTERN).foreach { excludePattern =>
+ val setRolledLogsExcludePatternMethod =
+ logAggregationContext.getClass.getMethod("setRolledLogsExcludePattern", classOf[String])
+ setRolledLogsExcludePatternMethod.invoke(logAggregationContext, excludePattern)
+ }
+
+ appContext.setLogAggregationContext(logAggregationContext)
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Ignoring ${ROLLED_LOG_INCLUDE_PATTERN.key} because the version of YARN " +
+ "does not support it", e)
}
- appContext.setLogAggregationContext(logAggregationContext)
}
appContext
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org