You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2022/06/20 08:30:50 UTC
[spark] branch master updated: [SPARK-39491][YARN] Fix yarn module compilation error with `-Phadoop-2` profile
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 1128cc08132 [SPARK-39491][YARN] Fix yarn module compilation error with `-Phadoop-2` profile
1128cc08132 is described below
commit 1128cc08132383ad35e383644f3b378d930b6c4c
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Mon Jun 20 17:30:34 2022 +0900
[SPARK-39491][YARN] Fix yarn module compilation error with `-Phadoop-2` profile
### What changes were proposed in this pull request?
Build `yarn` module with `-Phadoop-2` profile failed now as follows:
```
[ERROR] [Error] /basedir/spark/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:454: value DECOMMISSIONING is not a member of object org.apache.hadoop.yarn.api.records.NodeState
```
The above compilation error due to `hadoop-2.7` not support `NodeState.DECOMMISSIONING`, so this pr change to use string comparison instead for compilation, and the test suite `Test YARN container decommissioning` in `YarnAllocatorSuite` should only run when `VersionUtils.isHadoop3` is true
### Why are the changes needed?
Fix yarn module compilation error with `-Phadoop-2` profile
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GA
- Manual test
run `mvn clean install -DskipTests -pl resource-managers/yarn -am -Pyarn -Phadoop-2`
**Before**
```
[ERROR] [Error] /basedir/spark-source/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:454: value DECOMMISSIONING is not a member of object org.apache.hadoop.yarn.api.records.NodeState
[ERROR] one error found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.4.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 3.252 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 5.735 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 5.492 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 8.251 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 6.334 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 15.326 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 4.905 s]
[INFO] Spark Project Core ................................. SUCCESS [02:07 min]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 17.382 s]
[INFO] Spark Project YARN ................................. FAILURE [ 7.718 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:22 min
[INFO] Finished at: 2022-06-20T11:57:54+08:00
[INFO] ------------------------------------------------------------------------
```
**After**
```
[INFO] Reactor Summary for Spark Project Parent POM 3.4.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 5.451 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 5.739 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 5.908 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 8.310 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 5.857 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 8.439 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 4.795 s]
[INFO] Spark Project Core ................................. SUCCESS [02:36 min]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 15.044 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 32.517 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:09 min
[INFO] Finished at: 2022-06-20T13:10:04+08:00
[INFO] ------------------------------------------------------------------------
```
run `mvn clean install -pl resource-managers/yarn -Pyarn -Phadoop-2 -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnAllocatorSuite`
```
- Test YARN container decommissioning !!! CANCELED !!!
org.apache.spark.util.VersionUtils.isHadoop3 was false (YarnAllocatorSuite.scala:749)
Run completed in 2 seconds, 140 milliseconds.
Total number of tests run: 16
Suites: completed 2, aborted 0
Tests: succeeded 16, failed 0, canceled 6, ignored 0, pending 0
All tests passed.
```
Closes #36890
Closes #36917 from LuciferYang/SPARK-39491.
Lead-authored-by: yangjie01 <ya...@baidu.com>
Co-authored-by: Abhishek Dixit <ab...@gmail.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
.../src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala | 6 +++++-
.../scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala | 5 +++--
2 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
index 343f20fc94d..6158fbf8cd1 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
@@ -451,7 +451,11 @@ private[yarn] class YarnAllocator(
// resources on those nodes for earlier allocateResource calls, so notifying driver
// to put those executors in decommissioning state
allocateResponse.getUpdatedNodes.asScala.filter (node =>
- node.getNodeState == NodeState.DECOMMISSIONING &&
+ // SPARK-39491: Hadoop 2.7 does not support `NodeState.DECOMMISSIONING`,
+ // there change to use string comparison instead for compilation.
+ // Should revert to `node.getNodeState == NodeState.DECOMMISSIONING` when
+ // Hadoop 2.7 is no longer supported.
+ node.getNodeState.toString.equals("DECOMMISSIONING") &&
!decommissioningNodesCache.containsKey(getHostAddress(node)))
.foreach { node =>
val host = getHostAddress(node)
diff --git a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala
index c33a10f2e1c..2c7d7ac26c5 100644
--- a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala
+++ b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala
@@ -48,7 +48,7 @@ import org.apache.spark.resource.TestResourceIDs._
import org.apache.spark.rpc.RpcEndpointRef
import org.apache.spark.scheduler.SplitInfo
import org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.DecommissionExecutorsOnHost
-import org.apache.spark.util.ManualClock
+import org.apache.spark.util.{ManualClock, VersionUtils}
class MockResolver extends SparkRackResolver(SparkHadoopUtil.get.conf) {
@@ -746,6 +746,7 @@ class YarnAllocatorSuite extends SparkFunSuite with Matchers with BeforeAndAfter
}
test("Test YARN container decommissioning") {
+ assume(VersionUtils.isHadoop3)
val rmClient: AMRMClient[ContainerRequest] = AMRMClient.createAMRMClient()
val rmClientSpy = spy(rmClient)
val allocateResponse = mock(classOf[AllocateResponse])
@@ -794,7 +795,7 @@ class YarnAllocatorSuite extends SparkFunSuite with Matchers with BeforeAndAfter
// host1 is now in DECOMMISSIONING state
val httpAddress1 = "host1:420"
- when(nodeReport.getNodeState).thenReturn(NodeState.DECOMMISSIONING)
+ when(nodeReport.getNodeState).thenReturn(NodeState.valueOf("DECOMMISSIONING"))
when(nodeReport.getNodeId).thenReturn(nodeId)
when(nodeId.getHost).thenReturn("host1")
when(allocateResponse.getUpdatedNodes).thenReturn(nodeReportList)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org