You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by witgo <gi...@git.apache.org> on 2014/05/03 16:05:21 UTC
[GitHub] spark pull request: The default version of yarn is equal to the ha...
GitHub user witgo opened a pull request:
https://github.com/apache/spark/pull/626
The default version of yarn is equal to the hadoop version
This is a part of [PR 590](https://github.com/apache/spark/pull/590)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/witgo/spark yarn_version
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/626.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #626
----
commit c76763b875beedba0a144efe1d3b814cfc8b811b
Author: witgo <wi...@qq.com>
Date: 2014-05-03T13:57:09Z
The default value of yarn.version is equal to hadoop.version
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/626#discussion_r12259828
--- Diff: pom.xml ---
@@ -558,65 +560,8 @@
<artifactId>jets3t</artifactId>
<version>0.7.1</version>
</dependency>
- <dependency>
--- End diff --
Ah I see. I didn't understand what you were saying before. Is the issue just that the dependency resolution fails?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/626#discussion_r12259904
--- Diff: pom.xml ---
@@ -558,65 +560,8 @@
<artifactId>jets3t</artifactId>
<version>0.7.1</version>
</dependency>
- <dependency>
--- End diff --
I tested the current environment ,when leaving the yarn dependency declarations as they are now,` mvn -DskipTests clean package` can work.I will restore modified
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/626#discussion_r12259307
--- Diff: pom.xml ---
@@ -558,65 +560,8 @@
<artifactId>jets3t</artifactId>
<version>0.7.1</version>
</dependency>
- <dependency>
--- End diff --
You're right. but in `mvn -Pyarn clean package`, the hadoop version 2.2.0
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/626#discussion_r12259437
--- Diff: pom.xml ---
@@ -558,65 +560,8 @@
<artifactId>jets3t</artifactId>
<version>0.7.1</version>
</dependency>
- <dependency>
--- End diff --
Could you explain what you think the problem is?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/626#discussion_r12258918
--- Diff: pom.xml ---
@@ -558,65 +560,8 @@
<artifactId>jets3t</artifactId>
<version>0.7.1</version>
</dependency>
- <dependency>
--- End diff --
Could you also remove these changes around restructuring the dependency locations? These need to be tested and verified separately.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/626#discussion_r12258922
--- Diff: pom.xml ---
@@ -558,65 +560,8 @@
<artifactId>jets3t</artifactId>
<version>0.7.1</version>
</dependency>
- <dependency>
--- End diff --
Bascially what I want here is a ~10 to 15 line pull request we can verify and merge in quickly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/626#issuecomment-42125591
Great - thanks for paring this down. I can merge it. Let's look at cleaning this up once we ship 1.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/626#discussion_r12259068
--- Diff: pom.xml ---
@@ -558,65 +560,8 @@
<artifactId>jets3t</artifactId>
<version>0.7.1</version>
</dependency>
- <dependency>
--- End diff --
This is not very good, otherwise the default value is 1.0.4
```scala
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-client</artifactId>
<version>1.0.4</version>
</dependency>
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/626#discussion_r12259452
--- Diff: pom.xml ---
@@ -558,65 +560,8 @@
<artifactId>jets3t</artifactId>
<version>0.7.1</version>
</dependency>
- <dependency>
--- End diff --
I don't mind requiring that someone set `-Phadoop.version=2.2.0` if they run a build with `-Pyarn`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/626#discussion_r12259085
--- Diff: pom.xml ---
@@ -558,65 +560,8 @@
<artifactId>jets3t</artifactId>
<version>0.7.1</version>
</dependency>
- <dependency>
--- End diff --
What's the problem exactly? We don't ever rely on the default version here, right? If someone tries to build with `-Pyarn` but they don't set the hadoop version to be higher - the build can fail.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/626#discussion_r12259635
--- Diff: pom.xml ---
@@ -558,65 +560,8 @@
<artifactId>jets3t</artifactId>
<version>0.7.1</version>
</dependency>
- <dependency>
--- End diff --
Leaving the dependency declarations as they are now, `mvn -DskipTests clean package` can not be executed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/626#discussion_r12259611
--- Diff: pom.xml ---
@@ -558,65 +560,8 @@
<artifactId>jets3t</artifactId>
<version>0.7.1</version>
</dependency>
- <dependency>
--- End diff --
In `mvn -DskipTests clean package` the dependency declarations of `hadoop-yarn-api`,`hadoop-yarn-common`,`hadoop-yarn-client` is no necessary
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/626#discussion_r12259852
--- Diff: pom.xml ---
@@ -558,65 +560,8 @@
<artifactId>jets3t</artifactId>
<version>0.7.1</version>
</dependency>
- <dependency>
--- End diff --
Yes, I'm sorry, my english is relatively bad.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/626#issuecomment-42105826
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/626
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/626#discussion_r12259715
--- Diff: pom.xml ---
@@ -558,65 +560,8 @@
<artifactId>jets3t</artifactId>
<version>0.7.1</version>
</dependency>
- <dependency>
--- End diff --
When `hadop.version` is 1.0.4 , `yarn.version` is also 1.0.4
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-client</artifactId>
<version>${yarn.version}</version>
</dependency>
is not correct
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/626#discussion_r12259340
--- Diff: pom.xml ---
@@ -558,65 +560,8 @@
<artifactId>jets3t</artifactId>
<version>0.7.1</version>
</dependency>
- <dependency>
--- End diff --
|maven| hadoop.version | yarn.version |
| :------------ |:---------------:|:-----:|
|`mvn -Pyarn -DskipTests clean package`|2.2.0|2.2.0|
|`mvn -Phadoop-0.23 -Pyarn-alpha -DskipTests clean package`|0.23.7|0.23.7
|`mvn -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.2.0 -DskipTests clean package`| 2.0.0-cdh4.2.0|2.0.0-cdh4.2.0
|`mvn -Phadoop-0.23 -Pyarn-alpha -Dhadoop.version=2.3.0 -Dyarn.version=0.23.7 -DskipTests clean package`|2.3.0|0.23.7|
|`mvn -DskipTests clean package`|1.0.4|not support|
|`mvn -Pyarn-alpha -Dyarn.version=0.23.7 -Dhadoop.version=1.0.4 -Phadoop-0.23 -DskipTests package`|1.0.4|0.23.7|
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: The default version of yarn is equal to the ha...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/626#discussion_r12259629
--- Diff: pom.xml ---
@@ -558,65 +560,8 @@
<artifactId>jets3t</artifactId>
<version>0.7.1</version>
</dependency>
- <dependency>
--- End diff --
Yes, I understand, but let's clean this up separately. Right now we are about to make a release, so we are only merging immediate critical bug fixes. I understand it's cleaner to have it the other way, but verifying the correctness of the refactoring will require running a bunch of test builds on our end, and we want to avoid that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---