You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by sujith71955 <gi...@git.apache.org> on 2018/08/23 10:50:15 UTC
[GitHub] spark pull request #22199: [SPARK-25073][SQL]When wild card is been used in ...
GitHub user sujith71955 opened a pull request:
https://github.com/apache/spark/pull/22199
[SPARK-25073][SQL]When wild card is been used in load command system
## What changes were proposed in this pull request?
When the yarn.nodemanager.resource.memory-mb or yarn.scheduler.maximum-allocation-mb memory assignment is insufficient, Spark always reports an error request to adjust yarn.scheduler.maximum-allocation-mb
even though in message it shows the memory value of yarn.nodemanager.resource.memory-mb,As the error Message is bit misleading to the user we can modify the same, We can keep the error message same as executor memory validation message.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sujith71955/spark maste_am_log
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22199.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22199
----
commit 9fe1a6232c4ed61ef67c9baf6e5aaa751a55f3fe
Author: s71955 <su...@...>
Date: 2018-08-23T22:24:10Z
[SPARK-25073][SQL]When wild card is been used in load command system is throwing analysis exception
## What changes were proposed in this pull request?
When the yarn.nodemanager.resource.memory-mb or yarn.scheduler.maximum-allocation-mb memory assignment is insufficient, Spark always reports an error request to adjust yarn.scheduler.maximum-allocation-mb
even though in message it shows the memory value of yarn.nodemanager.resource.memory-mb.As the error Message is bit misleading to the user we can modify the same, We can keep the error message same as
executor memory validation message.
## How was this patch tested?
Manually tested in hdfs-Yarn clustaer
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22199: [SPARK-25073][Yarn] AM and Executor Memory valida...
Posted by sujith71955 <gi...@git.apache.org>.
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22199#discussion_r212370528
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -338,13 +338,14 @@ private[spark] class Client(
throw new IllegalArgumentException(s"Required executor memory ($executorMemory" +
s"+$executorMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster! " +
"Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or " +
- "'yarn.nodemanager.resource.memory-mb'.")
+ "'yarn.nodemanager.resource.memory-mb and increase the memory appropriately.")
--- End diff --
Yes both settings has to be changed, but for the AM container memory allocation validation only yarn.scheduler.maximum-allocation-mb parameter is mentioned message does not indicate to increase yarn.nodemanager.resource.memory-mb, in my change i updated this message by including "yarn.nodemanager.resource.memory-mb" paramter
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22199: [SPARK-25073][Yarn] AM and Executor Memory valida...
Posted by sujith71955 <gi...@git.apache.org>.
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22199#discussion_r212392740
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -338,13 +338,14 @@ private[spark] class Client(
throw new IllegalArgumentException(s"Required executor memory ($executorMemory" +
s"+$executorMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster! " +
"Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or " +
- "'yarn.nodemanager.resource.memory-mb'.")
+ "'yarn.nodemanager.resource.memory-mb and increase the memory appropriately.")
--- End diff --
just trying to make the messages more informative by mentioning to increase the memory appropriately as per the business use-case .i shall remove the part if it doesn't make much difference to the information we are trying to convey.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22199: [SPARK-25073][SQL]When wild card is been used in load co...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22199
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22199: [SPARK-25073][Yarn] AM and Executor Memory validation me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22199
**[Test build #95196 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95196/testReport)** for PR 22199 at commit [`3f4051c`](https://github.com/apache/spark/commit/3f4051cd0895debe1c4feeee288890b397d5dfe1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22199: [SPARK-25073][Yarn] AM and Executor Memory valida...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/22199#discussion_r212301867
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -338,13 +338,14 @@ private[spark] class Client(
throw new IllegalArgumentException(s"Required executor memory ($executorMemory" +
s"+$executorMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster! " +
"Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or " +
- "'yarn.nodemanager.resource.memory-mb'.")
+ "'yarn.nodemanager.resource.memory-mb and increase the memory appropriately.")
--- End diff --
What does this add?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22199: [SPARK-25073][Yarn] AM and Executor Memory validation me...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22199
**[Test build #95196 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95196/testReport)** for PR 22199 at commit [`3f4051c`](https://github.com/apache/spark/commit/3f4051cd0895debe1c4feeee288890b397d5dfe1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22199: [SPARK-25073][Yarn] AM and Executor Memory validation me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22199
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22199: [SPARK-25073][Yarn] AM and Executor Memory valida...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/22199#discussion_r212389472
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -338,13 +338,14 @@ private[spark] class Client(
throw new IllegalArgumentException(s"Required executor memory ($executorMemory" +
s"+$executorMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster! " +
"Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or " +
- "'yarn.nodemanager.resource.memory-mb'.")
+ "'yarn.nodemanager.resource.memory-mb and increase the memory appropriately.")
--- End diff --
I understand the change below, that's probably fine. The change above doesn't seem to add anything. You want to tell the user to check both settings in both cases.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22199: [SPARK-25073][Yarn] AM and Executor Memory valida...
Posted by sujith71955 <gi...@git.apache.org>.
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22199#discussion_r212396099
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -338,13 +338,14 @@ private[spark] class Client(
throw new IllegalArgumentException(s"Required executor memory ($executorMemory" +
s"+$executorMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster! " +
"Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or " +
- "'yarn.nodemanager.resource.memory-mb'.")
+ "'yarn.nodemanager.resource.memory-mb and increase the memory appropriately.")
--- End diff --
Got your point, i will update as per the suggestion. Thanks for the feedback Sean.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22199: [SPARK-25073][Yarn] AM and Executor Memory valida...
Posted by sujith71955 <gi...@git.apache.org>.
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22199#discussion_r212356651
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -338,13 +338,14 @@ private[spark] class Client(
throw new IllegalArgumentException(s"Required executor memory ($executorMemory" +
s"+$executorMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster! " +
"Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or " +
- "'yarn.nodemanager.resource.memory-mb'.")
+ "'yarn.nodemanager.resource.memory-mb and increase the memory appropriately.")
--- End diff --
Please let me know for any suggestions. Thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22199: [SPARK-25073][Yarn] AM and Executor Memory valida...
Posted by sujith71955 <gi...@git.apache.org>.
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22199#discussion_r212356336
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -338,13 +338,14 @@ private[spark] class Client(
throw new IllegalArgumentException(s"Required executor memory ($executorMemory" +
s"+$executorMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster! " +
"Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or " +
- "'yarn.nodemanager.resource.memory-mb'.")
+ "'yarn.nodemanager.resource.memory-mb and increase the memory appropriately.")
--- End diff --
Where as in the executor memory validation check we are displaying the proper message considering both yarn.nodemanager.resource.memory-mb and yarn.scheduler.maximum-allocation-mb
```
if (executorMem > maxMem) {
throw new IllegalArgumentException(s"Required executor memory ($executorMemory" +
s"+$executorMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster! " +
"Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or " +
"'yarn.nodemanager.resource.memory-mb and increase the memory appropriately.")
}
```
so same message i think is required for am memory validation as well
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22199: [SPARK-25073][Yarn] AM and Executor Memory valida...
Posted by sujith71955 <gi...@git.apache.org>.
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22199#discussion_r212354862
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -338,13 +338,14 @@ private[spark] class Client(
throw new IllegalArgumentException(s"Required executor memory ($executorMemory" +
s"+$executorMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster! " +
"Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or " +
- "'yarn.nodemanager.resource.memory-mb'.")
+ "'yarn.nodemanager.resource.memory-mb and increase the memory appropriately.")
--- End diff --
As mentioned in the JIRA even though the memory defined in the yarn.nodemanager.resource.memory-mb parameter is less eg:
yarn.scheduler.maximum-allocation-mb =15g and yarn.nodemanager.resource.memory-mb =8g
Launch spark-shell --master yarn --conf spark.yarn.am.memory=10g
we are getting below error
java.lang.IllegalArgumentException: Required AM memory (10240+1024 MB) is above the max threshold (8096 MB) of this cluster! Please increase the value of 'yarn.scheduler.maximum-allocation-mb'.
This message is very confusing to a user since spark indicate to increase the arn.scheduler.maximum-allocation-mb which is actually more than 10G as per this scenario whereas the issue is with yarn.nodemanager.resource.memory-mb,
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22199: [SPARK-25073][Yarn] AM and Executor Memory validation me...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/22199
ok to test.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22199: [SPARK-25073][SQL]When wild card is been used in load co...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22199
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22199: [SPARK-25073][Yarn] AM and Executor Memory valida...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/22199
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22199: [SPARK-25073][SQL]When wild card is been used in load co...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22199
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22199: [SPARK-25073][Yarn] AM and Executor Memory validation me...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22199
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95196/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22199: [SPARK-25073][Yarn] AM and Executor Memory valida...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/22199#discussion_r212395269
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -338,13 +338,14 @@ private[spark] class Client(
throw new IllegalArgumentException(s"Required executor memory ($executorMemory" +
s"+$executorMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster! " +
"Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or " +
- "'yarn.nodemanager.resource.memory-mb'.")
+ "'yarn.nodemanager.resource.memory-mb and increase the memory appropriately.")
--- End diff --
Sure, though the solution may be to reduce the app memory usage rather than increase the YARN limits (which only admins can do). I think that's why the message more neutrally suggests that these are the limits that the app is hitting.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22199: [SPARK-25073][Yarn] AM and Executor Memory valida...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/22199#discussion_r212356312
--- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -338,13 +338,14 @@ private[spark] class Client(
throw new IllegalArgumentException(s"Required executor memory ($executorMemory" +
s"+$executorMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster! " +
"Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or " +
- "'yarn.nodemanager.resource.memory-mb'.")
+ "'yarn.nodemanager.resource.memory-mb and increase the memory appropriately.")
--- End diff --
Both settings may have to change. But your text change doesn't mention anything new. Why does this help? I also don't think that the current behavior is wrong
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org