You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/06/03 20:51:53 UTC
[GitHub] [spark] gerashegalov opened a new pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
gerashegalov opened a new pull request #28720:
URL: https://github.com/apache/spark/pull/28720
An example of a command that fail in the client mode but would not in the cluster mode or if the same value were used for `spark.executor.memory`
```
> $SPARK_HOME/bin/spark-shell --conf spark.driver.memory=" 4g "
Invalid maximum heap size: -Xmx 4g
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
```
also note that the same message is produced by the JVM makes it hard to identify which of the configs SPARK_DAEMON_MEMORY, SPARK_DRIVER_MEMORY, etc was responsible
### Why are the changes needed?
easier ops/diagnostics, more uniform behavior
### Does this PR introduce _any_ user-facing change?
fix-only change
### How was this patch tested?
- unit tests
- manual testing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-638455734
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-646402044
**[Test build #124240 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124240/testReport)** for PR 28720 at commit [`87e1d67`](https://github.com/apache/spark/commit/87e1d67c5be394ae514e38a50958f88ecc721287).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-646355753
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-638456255
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-639197112
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-639147791
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-639149691
**[Test build #123546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123546/testReport)** for PR 28720 at commit [`87e1d67`](https://github.com/apache/spark/commit/87e1d67c5be394ae514e38a50958f88ecc721287).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-639147436
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gerashegalov commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
gerashegalov commented on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-700789365
thanks for looking at the PR @srowen. Since there are multiple culprits potentially responsible for the issue I thought a patch was warrant. And I like to see consistent behavior between similar settings. However, I also understand the tradeoff of accepting more code for future maintenance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-646402640
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-639196496
**[Test build #123546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123546/testReport)** for PR 28720 at commit [`87e1d67`](https://github.com/apache/spark/commit/87e1d67c5be394ae514e38a50958f88ecc721287).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-646355461
**[Test build #124240 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124240/testReport)** for PR 28720 at commit [`87e1d67`](https://github.com/apache/spark/commit/87e1d67c5be394ae514e38a50958f88ecc721287).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-646355753
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-638455734
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-639147791
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
srowen closed pull request #28720:
URL: https://github.com/apache/spark/pull/28720
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-699714183
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-646355161
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-646355461
**[Test build #124240 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124240/testReport)** for PR 28720 at commit [`87e1d67`](https://github.com/apache/spark/commit/87e1d67c5be394ae514e38a50958f88ecc721287).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-638456255
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-639197112
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-639149691
**[Test build #123546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123546/testReport)** for PR 28720 at commit [`87e1d67`](https://github.com/apache/spark/commit/87e1d67c5be394ae514e38a50958f88ecc721287).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28720: [SPARK-31900][SPARK-SUBMIT] Client memory passed unvalidated to the JVM Xmx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28720:
URL: https://github.com/apache/spark/pull/28720#issuecomment-646402640
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org