You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by srowen <gi...@git.apache.org> on 2016/08/11 10:42:57 UTC
[GitHub] spark pull request #14602: [SPARK-16966] [SQL] [CORE] App Name is a randomUU...
GitHub user srowen opened a pull request:
https://github.com/apache/spark/pull/14602
[SPARK-16966] [SQL] [CORE] App Name is a randomUUID even when "spark.app.name" exists
## What changes were proposed in this pull request?
Don't override spark.app.name in SparkSession when making a new context, as it will override spark.app.name in SparkConf, and will have already been checked anyway by SparkSubmit
## How was this patch tested?
Jenkins tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/srowen/spark SPARK-16966
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14602.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14602
----
commit 94570ae3b03aad1fa55b4148633b6bab3b829722
Author: Sean Owen <so...@cloudera.com>
Date: 2016-08-11T10:42:01Z
Don't override spark.app.name in SparkSession when making a new context, as it will override spark.app.name in SparkConf, and will have already been checked anyway by SparkSubmit
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14602: [SPARK-16966] [SQL] [CORE] App Name is a randomUUID even...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14602
**[Test build #63610 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63610/consoleFull)** for PR 14602 at commit [`e3e8929`](https://github.com/apache/spark/commit/e3e8929107a503ee6638fa780e30d360c472e0c4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14602: [SPARK-16966] [SQL] [CORE] App Name is a randomUUID even...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14602
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14602: [SPARK-16966] [SQL] [CORE] App Name is a randomUUID even...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14602
**[Test build #63607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63607/consoleFull)** for PR 14602 at commit [`94570ae`](https://github.com/apache/spark/commit/94570ae3b03aad1fa55b4148633b6bab3b829722).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14602: [SPARK-16966] [SQL] [CORE] App Name is a randomUUID even...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/14602
The thing is, we've always set an app name in all tests and examples, so I have presumed that it's required even if running without spark-submit. I'd assume the same behavior is assumed for SparkSession. In fact, all the code does set app name, except the one test suite. I assumed it might have been a quick band-aid and not the best solution. Yes you can also check the conf for spark.app.name; it just seemed funny to intro new behavior and another different place and way to set a default name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14602: [SPARK-16966] [SQL] [CORE] App Name is a randomUUID even...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14602
**[Test build #63607 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63607/consoleFull)** for PR 14602 at commit [`94570ae`](https://github.com/apache/spark/commit/94570ae3b03aad1fa55b4148633b6bab3b829722).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14602: [SPARK-16966] [SQL] [CORE] App Name is a randomUUID even...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14602
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63607/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14602: [SPARK-16966] [SQL] [CORE] App Name is a randomUUID even...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14602
**[Test build #63610 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63610/consoleFull)** for PR 14602 at commit [`e3e8929`](https://github.com/apache/spark/commit/e3e8929107a503ee6638fa780e30d360c472e0c4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14602: [SPARK-16966] [SQL] [CORE] App Name is a randomUUID even...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14602
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63610/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14602: [SPARK-16966] [SQL] [CORE] App Name is a randomUUID even...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14602
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14602: [SPARK-16966] [SQL] [CORE] App Name is a randomUU...
Posted by srowen <gi...@git.apache.org>.
Github user srowen closed the pull request at:
https://github.com/apache/spark/pull/14602
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14602: [SPARK-16966] [SQL] [CORE] App Name is a randomUUID even...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14602
Hmm... not all apps start via spark-submit; so it is possible to create a SparkSession before setting the app's name; in which case the `SparkContext.getOrCreate` call (a few lines below your change to SparkSession.scala) could fail if no context currently exists.
Instead, shouldn't the code use `sparkConf.setIfMissing` after the SparkConf is instantiated?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14602: [SPARK-16966] [SQL] [CORE] App Name is a randomUUID even...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/14602
I'm personally fine with "things break if you don't set app name", but that's not the expectation set in the API:
```
/**
* Sets a name for the application, which will be shown in the Spark web UI.
* If no application name is set, a randomly generated name will be used.
*
* @since 2.0.0
*/
def appName(name: String): Builder = config("spark.app.name", name)
```
So technically that would be breaking the current API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14602: [SPARK-16966] [SQL] [CORE] App Name is a randomUUID even...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/14602
True, darn. If it's been promised as behavior, I think we'd have to keep the behavior. OK, backing up ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org