You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dajac <gi...@git.apache.org> on 2014/10/16 08:50:45 UTC
[GitHub] spark pull request: SPARK-3965: Ensure that Spark assembly for had...
GitHub user dajac opened a pull request:
https://github.com/apache/spark/pull/2822
SPARK-3965: Ensure that Spark assembly for hadoop2 contains avro-mapred for hadoop2
When building current Spark assembly for hadoop2, the wrong version of avro-mapred is picked. This patch adds org.apache.avro:avro-mapred as direct dependency of the assembly to ensure that the correct version is added to it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dajac/spark SPARK-3965
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2822.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2822
----
commit be820a27d9efab62b70486062db79ebcbdb41301
Author: David Jacot <da...@gmail.com>
Date: 2014-10-16T05:38:05Z
SPARK-3965: Ensure that Spark assembly for hadoop2 contains avro-mapred for hadoop2 when assembly is built.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2822#issuecomment-59456867
Thanks this seems like a good catch - just so we can reproduce the issue, can you provide the exact command you were running to build for Hadoop2?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...
Posted by goosea <gi...@git.apache.org>.
Github user goosea commented on the pull request:
https://github.com/apache/spark/pull/2822#issuecomment-63444896
avro-mapred 1.7.5 (no classifier) is coming from org.spark-project.hive_hive-exec:jar:0.13.1a , overriding avro-mapred 1.7.6-hadoop2
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...
Posted by dajac <gi...@git.apache.org>.
Github user dajac closed the pull request at:
https://github.com/apache/spark/pull/2822
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-3965: Ensure that Spark assembly for had...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2822#issuecomment-59321034
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...
Posted by dajac <gi...@git.apache.org>.
Github user dajac commented on the pull request:
https://github.com/apache/spark/pull/2822#issuecomment-59478665
I used following command:
```bash
mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive -DskipTests clean package
```
You'll see that org.apache.avro:avro-mapred:jar is picked but it should be org.apache.avro:avro-mapred:jar:hadoop2.
The issue only happens when you bring hive module in the assembly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/2822#issuecomment-73410935
This should already be resolved by https://github.com/apache/spark/commit/75fdccca32972f86a975033d7c4ce576dd79290f / SPARK-3039. Do you mind closing this? or, comment if you think this didn't resolve the issue you have in mind.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...
Posted by dajac <gi...@git.apache.org>.
Github user dajac commented on the pull request:
https://github.com/apache/spark/pull/2822#issuecomment-61469324
@pwendell any news on this issue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2822#issuecomment-66537561
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2822#issuecomment-59457425
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21831/consoleFull) for PR 2822 at commit [`be820a2`](https://github.com/apache/spark/commit/be820a27d9efab62b70486062db79ebcbdb41301).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2822#issuecomment-59457051
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...
Posted by dajac <gi...@git.apache.org>.
Github user dajac commented on the pull request:
https://github.com/apache/spark/pull/2822#issuecomment-73419336
Indeed, it looks OK. I close it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...
Posted by jbeynon <gi...@git.apache.org>.
Github user jbeynon commented on the pull request:
https://github.com/apache/spark/pull/2822#issuecomment-63406837
I still see this issue with the 1.2.0 preview release that Patrick posted this morning.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2822#issuecomment-66537834
[Test build #24328 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24328/consoleFull) for PR 2822 at commit [`be820a2`](https://github.com/apache/spark/commit/be820a27d9efab62b70486062db79ebcbdb41301).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2822#issuecomment-59461735
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21831/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...
Posted by dajac <gi...@git.apache.org>.
Github user dajac commented on the pull request:
https://github.com/apache/spark/pull/2822#issuecomment-63432550
@pwendell It would be great to have this one in 1.2.0 to make deployment on YARN easier.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2822#issuecomment-59461732
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21831/consoleFull) for PR 2822 at commit [`be820a2`](https://github.com/apache/spark/commit/be820a27d9efab62b70486062db79ebcbdb41301).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2822#issuecomment-60014085
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org