You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by li-zhihui <gi...@git.apache.org> on 2014/07/31 09:27:06 UTC
[GitHub] spark pull request: Fix conflict between code and doc in YarnClien...
GitHub user li-zhihui opened a pull request:
https://github.com/apache/spark/pull/1684
Fix conflict between code and doc in YarnClientSchedulerBackend.scala
Doc say: system properties override environment variables.
https://github.com/apache/spark/blob/master/yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala#L71
But code is conflict with it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/li-zhihui/spark fixaddarg
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1684.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1684
----
commit f1be1f01bc86abb3b86461b49842f704e262d279
Author: Li Zhihui <zh...@intel.com>
Date: 2014-07-31T06:27:30Z
Fix conflict between code and doc in YarnClientSchedulerBackend.scala
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Fix conflict between code and doc in YarnClien...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-51284115
@li-zhihui good point about YarnClusterSchedulerBackend, Taking a deeper look there is YarnClusterSchedulerBackend and then there are a couple other places in which the conf overrides the ENV variable. So lets keep it consistent and go with your original change. Sorry for the extra work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2872] Fix conflict between code and doc...
Posted by li-zhihui <gi...@git.apache.org>.
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-52873228
@tgravescs
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2872] Fix conflict between code and doc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-51333176
QA tests have started for PR 1684. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18020/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2872] Fix conflict between code and doc...
Posted by li-zhihui <gi...@git.apache.org>.
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-51285402
@tgravescs done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fix conflict between code and doc in YarnClien...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-51220306
At this point we have released it with env variable overriding configs. I think it would be better to just update the comment (since its just in the code and not user facing) to say what it does vs changing the user behavior.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fix conflict between code and doc in YarnClien...
Posted by li-zhihui <gi...@git.apache.org>.
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-51282316
@tgravescs got it.
But, <code>YarnClusterSchedulerBackend</code> use the same strategy (code and comment).
https://github.com/apache/spark/blob/master/yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerBackend.scala#L41
I think we should fix it to env variable overriding config too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2872] Fix conflict between code and doc...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-52936217
sorry I haven't had time to get back to this. Originally there were some issues with the order and make sure all the options worked properly from all the various ways to submit - directly using spark-class and then using spark-submit. I believe one of them was the SPARK_YARN_APP_NAME.
Can you test to make sure they all work, see https://github.com/apache/spark/pull/539.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2872] Fix conflict between code and doc...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-55408934
Note this is going to conflict with https://github.com/apache/spark/pull/2350
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2872] Fix conflict between code and doc...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-51332852
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2872] Fix conflict between code and doc...
Posted by li-zhihui <gi...@git.apache.org>.
Github user li-zhihui closed the pull request at:
https://github.com/apache/spark/pull/1684
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2872] Fix conflict between code and doc...
Posted by li-zhihui <gi...@git.apache.org>.
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-51420203
@tgravescs
Test failure because " FileServerSuite:Build timed out (after 120 minutes)", but I guess the patch is irrelevant to the issue.
Can you ask jenkins test this again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fix conflict between code and doc in YarnClien...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-50725105
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Fix conflict between code and doc in YarnClien...
Posted by li-zhihui <gi...@git.apache.org>.
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-51283355
@tgravescs I rollback previous commit and add a new commit just update comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fix conflict between code and doc in YarnClien...
Posted by li-zhihui <gi...@git.apache.org>.
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-50837342
@tgravescs can you please to have a look at this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2872] Fix conflict between code and doc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-54694540
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fix conflict between code and doc in YarnClien...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-51284671
can you also please file a jira for this and link the PR to it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org