You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/06/16 17:44:50 UTC
[GitHub] [spark] abhishekd0907 opened a new pull request, #36890: [SPARK-30835][SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
abhishekd0907 opened a new pull request, #36890:
URL: https://github.com/apache/spark/pull/36890
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
Sprk build fails with hadoop-2 profile, which uses hadoop 2.7 version fails due to:
`
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:454: value DECOMMISSIONING is not a member of object org.apache.hadoop.yarn.api.records.NodeState
`
This is because DECOMMISSIONING was only added in hadoop 2.8.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
To fix Hadoop 2.7 Build
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
Existing unit tests should work fine with all versions of Hadoop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #36890: [SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #36890:
URL: https://github.com/apache/spark/pull/36890#issuecomment-1159466129
> @HyukjinKwon I test manually as follows without this pr:
>
> ```
> mvn clean install -DskipTests -pl resource-managers/yarn -am -Pyarn -Phadoop-2
> ```
>
> ```
> [INFO] Compiling 25 Scala sources to /basedir/spark/resource-managers/yarn/target/scala-2.12/classes ...
> [ERROR] [Error] /basedir/spark/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:454: value DECOMMISSIONING is not a member of object org.apache.hadoop.yarn.api.records.NodeState
> [ERROR] one error found
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary for Spark Project Parent POM 3.4.0-SNAPSHOT:
> [INFO]
> [INFO] Spark Project Parent POM ........................... SUCCESS [ 3.637 s]
> [INFO] Spark Project Tags ................................. SUCCESS [ 10.738 s]
> [INFO] Spark Project Local DB ............................. SUCCESS [ 5.492 s]
> [INFO] Spark Project Networking ........................... SUCCESS [ 8.388 s]
> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 5.692 s]
> [INFO] Spark Project Unsafe ............................... SUCCESS [ 7.937 s]
> [INFO] Spark Project Launcher ............................. SUCCESS [ 4.376 s]
> [INFO] Spark Project Core ................................. SUCCESS [02:56 min]
> [INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 24.754 s]
> [INFO] Spark Project YARN ................................. FAILURE [ 12.485 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 04:20 min
> [INFO] Finished at: 2022-06-18T20:18:16+08:00
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.3.0:compile (scala-compile-first) on project spark-yarn_2.12: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.3.0:compile failed.: CompileFailed -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please read the following articles:
> [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the command
> [ERROR] mvn <args> -rf :spark-yarn_2.12
> ```
with this pr build success
```
[INFO] Reactor Summary for Spark Project Parent POM 3.4.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 3.747 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 13.542 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 13.811 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 17.388 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 9.835 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 11.792 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 9.364 s]
[INFO] Spark Project Core ................................. SUCCESS [03:02 min]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 30.487 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 33.975 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:27 min
[INFO] Finished at: 2022-06-18T21:32:30+08:00
[INFO] ------------------------------------------------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] abhishekd0907 commented on pull request #36890: [SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
abhishekd0907 commented on PR #36890:
URL: https://github.com/apache/spark/pull/36890#issuecomment-1160122533
> Any updates of this PR?
@LuciferYang I have updated the PR as per your comments and your related PR #36917 . Sorry for the delay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #36890: [SPARK-30835][SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #36890:
URL: https://github.com/apache/spark/pull/36890#issuecomment-1158531743
Hm, but we lately release it with Hadoop 2.7 build though. which command did you use to build?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #36890: [SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #36890:
URL: https://github.com/apache/spark/pull/36890#issuecomment-1159906602
Any updates of this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #36890: [SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #36890:
URL: https://github.com/apache/spark/pull/36890#discussion_r900920200
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:
##########
@@ -1004,3 +1020,35 @@ private object YarnAllocator {
ContainerExitStatus.DISKS_FAILED
)
}
+
+/**
+ * State of the node.
+ * Add the node state depending upon the cluster manager for Yarn
+ */
+private[spark] object NodeState extends Enumeration {
+ val RUNNING, DECOMMISSIONED, DECOMMISSIONING, LOST, OTHER = Value
+ type NodeState = Value
+
+ // Helper method to get NodeState of the Yarn.
+ def getYarnNodeState(state: YarnNodeState): NodeState.Value = {
+ // In hadoop-2.7 there is no support for node state DECOMMISSIONING
+ // In Hadoop-2.8, hadoop3.1 and later version of spark there is a support
+ // to node state DECOMMISSIONING.
+ // Inorder to build the spark using hadoop2 and hadoop3,
+ // using string comparison for the YARN node state DECOMMISSIONING here and
+ // for other states we are matching the YarnNodeState and assigning
+ // the node state at spark end
+ if (state.toString.equals(NodeState.DECOMMISSIONING.toString)) {
Review Comment:
should we change to
```scala
state match {
case YarnNodeState.RUNNING => NodeState.RUNNING
case YarnNodeState.DECOMMISSIONED => NodeState.DECOMMISSIONED
case YarnNodeState.LOST => NodeState.LOST
case YarnNodeState.UNHEALTHY => NodeState.LOST
case other if other.toString.equals(NodeState.DECOMMISSIONING.toString) =>
NodeState.DECOMMISSIONING
case _ => NodeState.OTHER
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #36890: [SPARK-30835][SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #36890:
URL: https://github.com/apache/spark/pull/36890#issuecomment-1159443559
[SPARK-30835](https://issues.apache.org/jira/browse/SPARK-30835) already RESOLVED
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] abhishekd0907 commented on pull request #36890: [SPARK-30835][SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
abhishekd0907 commented on PR #36890:
URL: https://github.com/apache/spark/pull/36890#issuecomment-1159350112
@tgravescs @mridulm @attilapiros @HyukjinKwon @LuciferYang
please review,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #36890: [SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #36890:
URL: https://github.com/apache/spark/pull/36890#issuecomment-1159455357
@HyukjinKwon I test manually as follows:
```
mvn clean install -DskipTests -pl resource-managers/yarn -am -Pyarn -Phadoop-2
```
```
[INFO] Compiling 25 Scala sources to /basedir/spark/resource-managers/yarn/target/scala-2.12/classes ...
[ERROR] [Error] /basedir/spark/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:454: value DECOMMISSIONING is not a member of object org.apache.hadoop.yarn.api.records.NodeState
[ERROR] one error found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.4.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 3.637 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 10.738 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 5.492 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 8.388 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 5.692 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 7.937 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 4.376 s]
[INFO] Spark Project Core ................................. SUCCESS [02:56 min]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 24.754 s]
[INFO] Spark Project YARN ................................. FAILURE [ 12.485 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:20 min
[INFO] Finished at: 2022-06-18T20:18:16+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.3.0:compile (scala-compile-first) on project spark-yarn_2.12: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.3.0:compile failed.: CompileFailed -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :spark-yarn_2.12
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #36890: [SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #36890:
URL: https://github.com/apache/spark/pull/36890#discussion_r900916864
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:
##########
@@ -462,6 +463,21 @@ private[yarn] class YarnAllocator(
private def getHostAddress(nodeReport: NodeReport): String = nodeReport.getNodeId.getHost
+ /**
+ * Checks if node is in DECOMMISSIONING STATE
+ *
+ * Visible for testing.
+ */
+ def isDecommissioningNode(node: NodeReport): Boolean = {
Review Comment:
Can we change this to `private` and use `invokePrivate` in test suite?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #36890: [SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #36890:
URL: https://github.com/apache/spark/pull/36890#issuecomment-1159465095
> @HyukjinKwon I test manually as follows without this pr:
>
> ```
> mvn clean install -DskipTests -pl resource-managers/yarn -am -Pyarn -Phadoop-2
> ```
>
> ```
> [INFO] Compiling 25 Scala sources to /basedir/spark/resource-managers/yarn/target/scala-2.12/classes ...
> [ERROR] [Error] /basedir/spark/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:454: value DECOMMISSIONING is not a member of object org.apache.hadoop.yarn.api.records.NodeState
> [ERROR] one error found
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary for Spark Project Parent POM 3.4.0-SNAPSHOT:
> [INFO]
> [INFO] Spark Project Parent POM ........................... SUCCESS [ 3.637 s]
> [INFO] Spark Project Tags ................................. SUCCESS [ 10.738 s]
> [INFO] Spark Project Local DB ............................. SUCCESS [ 5.492 s]
> [INFO] Spark Project Networking ........................... SUCCESS [ 8.388 s]
> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 5.692 s]
> [INFO] Spark Project Unsafe ............................... SUCCESS [ 7.937 s]
> [INFO] Spark Project Launcher ............................. SUCCESS [ 4.376 s]
> [INFO] Spark Project Core ................................. SUCCESS [02:56 min]
> [INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 24.754 s]
> [INFO] Spark Project YARN ................................. FAILURE [ 12.485 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 04:20 min
> [INFO] Finished at: 2022-06-18T20:18:16+08:00
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.3.0:compile (scala-compile-first) on project spark-yarn_2.12: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.3.0:compile failed.: CompileFailed -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please read the following articles:
> [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the command
> [ERROR] mvn <args> -rf :spark-yarn_2.12
> ```
Fortunately, v3.3 BUILD SUCCESS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #36890: [SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #36890: [SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
URL: https://github.com/apache/spark/pull/36890
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] abhishekd0907 commented on pull request #36890: [SPARK-30835][SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
abhishekd0907 commented on PR #36890:
URL: https://github.com/apache/spark/pull/36890#issuecomment-1159349944
> Hm, but we lately release it with Hadoop 2.7 build though. which command did you use to build?
@tgravescs since you filed the JIRA, can you please share the command you were using?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #36890: [SPARK-30835][SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #36890:
URL: https://github.com/apache/spark/pull/36890#issuecomment-1158430823
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #36890: [SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #36890:
URL: https://github.com/apache/spark/pull/36890#issuecomment-1159455494
Let me check with this pr later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #36890: [SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #36890:
URL: https://github.com/apache/spark/pull/36890#discussion_r900916864
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:
##########
@@ -462,6 +463,21 @@ private[yarn] class YarnAllocator(
private def getHostAddress(nodeReport: NodeReport): String = nodeReport.getNodeId.getHost
+ /**
+ * Checks if node is in DECOMMISSIONING STATE
+ *
+ * Visible for testing.
+ */
+ def isDecommissioningNode(node: NodeReport): Boolean = {
Review Comment:
Is it possible to make this method `private` and use `invokePrivate` in test suite?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #36890: [SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #36890:
URL: https://github.com/apache/spark/pull/36890#issuecomment-1160131842
Let me merge https://github.com/apache/spark/pull/36917 with adding @abhishekd0907 as a co-author since that approach was proposed in that Pr first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #36890: [SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #36890:
URL: https://github.com/apache/spark/pull/36890#issuecomment-1159445960
@abhishekd0907 how did you test this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #36890: [SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #36890:
URL: https://github.com/apache/spark/pull/36890#discussion_r900949722
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:
##########
@@ -462,6 +463,21 @@ private[yarn] class YarnAllocator(
private def getHostAddress(nodeReport: NodeReport): String = nodeReport.getNodeId.getHost
+ /**
+ * Checks if node is in DECOMMISSIONING STATE
+ *
+ * Visible for testing.
+ */
+ def isDecommissioningNode(node: NodeReport): Boolean = {
+ // In hadoop-2.7 there is no support for node state DECOMMISSIONING
+ // In Hadoop-2.8, hadoop3.1 and later version of spark there is a support
+ // to node state DECOMMISSIONING.
+ // Inorder to build the spark using hadoop2 and hadoop3, not
+ // using YarnNodeState for the node state DECOMMISSIONING here and
+ // instead comparing Node states on spark end
+ NodeState.getYarnNodeState(node.getNodeState) == NodeState.DECOMMISSIONING
Review Comment:
hmm... Is it really necessary to define `NodeState`?
Is
```
NodeState.getYarnNodeState(node.getNodeState) == NodeState.DECOMMISSIONING
```
equivalent to
```
node.getNodeState.toString.equals("DECOMMISSIONING")
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #36890: [SPARK-39491][YARN] Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #36890:
URL: https://github.com/apache/spark/pull/36890#discussion_r900951262
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:
##########
@@ -462,6 +463,21 @@ private[yarn] class YarnAllocator(
private def getHostAddress(nodeReport: NodeReport): String = nodeReport.getNodeId.getHost
+ /**
+ * Checks if node is in DECOMMISSIONING STATE
+ *
+ * Visible for testing.
+ */
+ def isDecommissioningNode(node: NodeReport): Boolean = {
+ // In hadoop-2.7 there is no support for node state DECOMMISSIONING
+ // In Hadoop-2.8, hadoop3.1 and later version of spark there is a support
+ // to node state DECOMMISSIONING.
+ // Inorder to build the spark using hadoop2 and hadoop3, not
+ // using YarnNodeState for the node state DECOMMISSIONING here and
+ // instead comparing Node states on spark end
+ NodeState.getYarnNodeState(node.getNodeState) == NodeState.DECOMMISSIONING
Review Comment:
It seems that just change line 454 from
```
node.getNodeState == NodeState.DECOMMISSIONING
```
to
```
node.getNodeState.toString.equals("DECOMMISSIONING")
```
and add some comments is enough
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org