You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by zjffdu <gi...@git.apache.org> on 2017/04/26 05:38:30 UTC
[GitHub] zeppelin pull request #2285: ZEPPELIN-2450. Move spark profile to pom file o...
GitHub user zjffdu opened a pull request:
https://github.com/apache/zeppelin/pull/2285
ZEPPELIN-2450. Move spark profile to pom file of parent project
### What is this PR for?
2 issues of spark profile:
* Duplicated profiles in spark and spark-dependencies
* The default profile can not be activated because it is in sub module. This cause spark module always use spark 2.0.2 in intellij even when I activate profile spark-2.1, this cause I fail to run `PySparkInterpreterTest` in intellij.
This PR move spark profiles to pom file of parent project. By default, profile spark-2.1 and scala-2.10 will be activated.
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2450
### How should this be tested?
CI pass
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-2450
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2285.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2285
----
commit 469ee91d4639d083e99c1ab63897e0df81df9e6f
Author: Jeff Zhang <zj...@apache.org>
Date: 2017-04-26T03:27:20Z
ZEPPELIN-2450. Move spark profile to pom file of parent project
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2285: ZEPPELIN-2450. Move spark profile to pom file of paren...
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul commented on the issue:
https://github.com/apache/zeppelin/pull/2285
@zjffdu All you mentioned are about advantage if we would have done, but I also think we have to consider negative effect for it. For instance, if someone want to move some of ignite profiles, which is less popular than spark, we don't have any reason not to allow it because spark already did. For the worst case, all of profiles can be located into root pom.xml. I totally agree that some popular - I don't know what we measure though - profiles locate in the root pom.xml is a bit helpful to extend new interpreter or other profiles but we should consider the negative effect.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2285: ZEPPELIN-2450. Move spark profile to pom file of paren...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2285
> we don't want interpreter's profile to locate in a root pom.xml because that means all other interpreters profiles can be located in a root
Can you explain more ? Shouldn't all modules share the same profile ? e.g. We have a new module A which also want to use spark-2.1 profile. Then for the current pom structure he has to copy spark-2.1 profile to his pom file. If he want to use a different spark 2.1 profile compared to spark module for his module, then he has to give it a different profile name. Otherwise it would conflict with spark 2.1 profile in spark module.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2285: ZEPPELIN-2450. Move spark profile to pom file o...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu closed the pull request at:
https://github.com/apache/zeppelin/pull/2285
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2285: ZEPPELIN-2450. Move spark profile to pom file o...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu closed the pull request at:
https://github.com/apache/zeppelin/pull/2285
---
[GitHub] zeppelin issue #2285: ZEPPELIN-2450. Move spark profile to pom file of paren...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/2285
+1 on what @jongyoul says
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2285: ZEPPELIN-2450. Move spark profile to pom file of paren...
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul commented on the issue:
https://github.com/apache/zeppelin/pull/2285
@zjffdu About this issue, moon and I discussed when we split `spark-dependencies` from `spark`. We don't want to make duplication you mentioned, on the contrary, we don't want interpreter's profile to locate in a root `pom.xml` because that means all other interpreters profiles can be located in a root. I know what you intend and how good it is, but can you find another solution for it? I really don't want to see all profiles of modules in a root.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2285: ZEPPELIN-2450. Move spark profile to pom file of paren...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2285
> in some other cases you might want spark interpreter to spark 2.2 and livy interpreter to spark 1.6
In that case, in both approaches we all have to create a new profile in livy module. And we can't name as spark-1.6, we need to give it a different name (like livy-spark-1.6). I mean when user specify profiles in maven build command, all the profiles in root module and sub moduels are in the same scope and they should have unique name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2285: ZEPPELIN-2450. Move spark profile to pom file of paren...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2285
@Leemoonsoo @felixcheung @jongyoul Please help review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2285: ZEPPELIN-2450. Move spark profile to pom file o...
Posted by zjffdu <gi...@git.apache.org>.
GitHub user zjffdu reopened a pull request:
https://github.com/apache/zeppelin/pull/2285
ZEPPELIN-2450. Move spark profile to pom file of parent project
### What is this PR for?
2 issues of spark profile:
* Duplicated profiles in spark and spark-dependencies
* The default profile can not be activated because it is in sub module. This cause spark module always use spark 2.0.2 in intellij even when I activate profile spark-2.1, this cause I fail to run `PySparkInterpreterTest` in intellij.
This PR move spark profiles to pom file of parent project. By default, profile spark-2.1 and scala-2.10 will be activated.
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2450
### How should this be tested?
CI pass
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-2450
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2285.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2285
----
commit 469ee91d4639d083e99c1ab63897e0df81df9e6f
Author: Jeff Zhang <zj...@apache.org>
Date: 2017-04-26T03:27:20Z
ZEPPELIN-2450. Move spark profile to pom file of parent project
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2285: ZEPPELIN-2450. Move spark profile to pom file o...
Posted by zjffdu <gi...@git.apache.org>.
GitHub user zjffdu reopened a pull request:
https://github.com/apache/zeppelin/pull/2285
ZEPPELIN-2450. Move spark profile to pom file of parent project
### What is this PR for?
2 issues of spark profile:
* Duplicated profiles in spark and spark-dependencies
* The default profile can not be activated because it is in sub module. This cause spark module always use spark 2.0.2 in intellij even when I activate profile spark-2.1, this cause I fail to run `PySparkInterpreterTest` in intellij.
This PR move spark profiles to pom file of parent project. By default, profile spark-2.1 and scala-2.10 will be activated.
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2450
### How should this be tested?
CI pass
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-2450
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2285.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2285
----
commit 50c07033ae608df557390226b0b230a7b7b69a1e
Author: Jeff Zhang <zj...@apache.org>
Date: 2017-04-26T03:27:20Z
ZEPPELIN-2450. Move spark profile to pom file of parent project
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2285: ZEPPELIN-2450. Move spark profile to pom file of paren...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/2285
spark might be unique in this - almost all other profile are used only within one module
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2285: ZEPPELIN-2450. Move spark profile to pom file of paren...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2285
I mean I don't see harm to move spark profile in root pom file, not to mention the benefit we could remove the duplication. I see spark project also put hadoop profile in root pom file although not all modules using hadoop profile.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2285: ZEPPELIN-2450. Move spark profile to pom file of paren...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/2285
I'd argue possibly both approaches are useful: in some cases you might want all interpreters against the same version of spark, in some other cases you might want spark interpreter to spark 2.2 and livy interpreter to spark 1.6
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2285: ZEPPELIN-2450. Move spark profile to pom file o...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu closed the pull request at:
https://github.com/apache/zeppelin/pull/2285
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---