You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Matthias Friedrich (JIRA)" <ji...@apache.org> on 2012/07/12 19:08:34 UTC
[jira] [Created] (CRUNCH-16) Fix dependency versions
Matthias Friedrich created CRUNCH-16:
----------------------------------------
Summary: Fix dependency versions
Key: CRUNCH-16
URL: https://issues.apache.org/jira/browse/CRUNCH-16
Project: Crunch
Issue Type: Task
Reporter: Matthias Friedrich
Priority: Critical
Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438976#comment-13438976 ]
Gabriel Reid commented on CRUNCH-16:
------------------------------------
I've realized that this is directly related to something else I've been working on with Crunch where I'm combining reflect and specific data, so I took a second look at doing automatic initialization of the CAN_COMBINE_SPECIFIC_AND_REFLECT_SCHEMAS.
As far as I know/recall, the relevant issue that was fixed in Avro 1.7.0 was AVRO-1046, which was about the ReflectDatumReader properly handling specific data; this was important because the ReflectDatumReader is used as the fallback anytime there is reflect data included.
The attached patch fixes a small bug in the existing patches (the ReflectDatumReader wouldn't be used if the CAN_COMBINE_SPECIFIC_AND_REFLECT_SCHEMAS bit was set), and it also includes code to dynamically initialize the CAN_COMBINE_SPECIFIC_AND_REFLECT_SCHEMAS (which is now final). Can you take a look and let me know what you think? I've run the integration tests on Avro 1.5.4, 1.6.2, and 1.7.0 and it works on all of them.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch, CRUNCH-16-avro-preparation.patch, CRUNCH-16-avro-reflect-specific-dynamic-init.patch.txt, CRUNCH-16-avro-reflect-specific-static-init.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439289#comment-13439289 ]
Josh Wills commented on CRUNCH-16:
----------------------------------
Ooh, one exception-- we need to swap in avro.version = 1.5.4 for the hadoop 2.0.0 build, putting us in a sort of weird position of using an older version of avro for a newer version of hadoop.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch, CRUNCH-16-avro-preparation.patch, CRUNCH-16-avro-reflect-specific-dynamic-init.patch.txt, CRUNCH-16-avro-reflect-specific-static-init.patch, CRUNCH-16-uber.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438800#comment-13438800 ]
Josh Wills commented on CRUNCH-16:
----------------------------------
Yeah, I spent some time grepping through the Avro APIs to find ways to distinguish between the 1.7.x and earlier versions, which is a less robust version of the same trick. I'm on that fence with you as to whether or not it's worth it, which I think means that it's probably not.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch, CRUNCH-16-avro-preparation.patch, CRUNCH-16-avro-reflect-specific-static-init.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438987#comment-13438987 ]
Josh Wills commented on CRUNCH-16:
----------------------------------
That's really cool-- let's go with your impl.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch, CRUNCH-16-avro-preparation.patch, CRUNCH-16-avro-reflect-specific-dynamic-init.patch.txt, CRUNCH-16-avro-reflect-specific-static-init.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439285#comment-13439285 ]
Gabriel Reid commented on CRUNCH-16:
------------------------------------
Ok, sounds good. Do we know if this means that this patch is fully ready now, or is there still stuff that needs to go in?
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch, CRUNCH-16-avro-preparation.patch, CRUNCH-16-avro-reflect-specific-dynamic-init.patch.txt, CRUNCH-16-avro-reflect-specific-static-init.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-16) Fix dependency versions
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated CRUNCH-16:
-----------------------------
Attachment: CRUNCH-16-uber.patch
This patch has your my Avro changes, Gabriel's dynamic initialization checks for Avro, Matthias' dependency cleanups, some additional exclusions I added to make our avro stuff map to the same exclusions hadoop-client uses, and some minor dependency scope changes. I think it's ready to go.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch, CRUNCH-16-avro-preparation.patch, CRUNCH-16-avro-reflect-specific-dynamic-init.patch.txt, CRUNCH-16-avro-reflect-specific-static-init.patch, CRUNCH-16-uber.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417312#comment-13417312 ]
Matthias Friedrich commented on CRUNCH-16:
------------------------------------------
Josh & Tom, my idea was to get this working with one version of Hadoop before adding more complexity. Maven profiles tend to make things a bit unwieldy.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Priority: Critical
> Labels: maven
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417053#comment-13417053 ]
Josh Wills commented on CRUNCH-16:
----------------------------------
A thought: this seems to dovetail w/CRUNCH-4, in that the dependencies we ship will vary based on the core hadoop package we're building/running against. The deps for hadoop 1.0.3 are different from hadoop 2.0.0-alpha are different from CDH4-2.0.0. It might be better to fix the profiles issue first and then fix this one, since we'll have to do it for every profile anyway.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Priority: Critical
> Labels: maven
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436309#comment-13436309 ]
Josh Wills commented on CRUNCH-16:
----------------------------------
I cringe at the thought of going back to 1.5.3. If anything, I'd like to bump our avro to 1.7.1.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418196#comment-13418196 ]
Matthias Friedrich commented on CRUNCH-16:
------------------------------------------
Yes, this will definitely work. However, it would make the build system more complicated and probably cause confusion for users. I asked on crunch-dev if anyone depends on smile; if not let's just remove it for now and keep things simple.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Priority: Critical
> Labels: maven
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418091#comment-13418091 ]
Josh Wills commented on CRUNCH-16:
----------------------------------
+1 for option 2, but send an email to the dev list and make sure no one is super-attached to it.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Priority: Critical
> Labels: maven
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417030#comment-13417030 ]
Rahul Sharma commented on CRUNCH-16:
------------------------------------
Few things that have come up while fixing this issue :
1. I think Jackson-smile did not exist before 1.6 version. This also depends on the 1.6 version of jackson-core-asl. But the platform bundles jackson-core-asl 1.5.2 version. So what should I do here ? The project currently uses 1.8.6 version of Jackson-smile .
2. There is a google-collections-guava lib that gets bundled on the platform. It comes in org.apache.hadoop.thirdparty package and is of version r09. The project currently uses 11.0.2 version of the same. Using it would mean correcting package structures through-out the application.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Priority: Critical
> Labels: maven
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-16) Fix dependency versions
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthias Friedrich updated CRUNCH-16:
-------------------------------------
Attachment: 0001-CRUNCH-16-Fix-dependency-versions.patch
Dependencies for crunch core on Hadoop 1 are OK now. HBase on Hadoop 1 still has conflicts with Jersey, but I hope the client doesn't need it. There's nothing we can do about it anyway. As far as I'm concerned, the hadoop-1 build profile is done.
The conflict between Hadoop 2's avro-1.5.3 and our 1.7.0 is currently unfixed. I'm not sure what to do about this: Downgrade avro, ignore the problem, or move avro to a separate maven module (which doesn't really solve the problem either). Ideas anyone?
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-16) Fix dependency versions
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gabriel Reid updated CRUNCH-16:
-------------------------------
Attachment: CRUNCH-16-avro-reflect-specific-dynamic-init.patch.txt
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch, CRUNCH-16-avro-preparation.patch, CRUNCH-16-avro-reflect-specific-dynamic-init.patch.txt, CRUNCH-16-avro-reflect-specific-static-init.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437264#comment-13437264 ]
Matthias Friedrich commented on CRUNCH-16:
------------------------------------------
Cool, that's probably the best we can do for Hadoop 2, thank you.
Regarding protobuf: The patch uses 2.3.0 for hadoop-1 because hbase-0.90.4 doesn't compile with 2.4.0a. It uses 2.4.0a for hadoop-2 because that's what Hadoop 2 has on its classpath and that's also what hbase-0.94.1 uses. Is there something I'm overlooking? I checked via "mvn dependency:tree" and "mvn dependency:tree -Phadoop-2".
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417118#comment-13417118 ]
Tom White commented on CRUNCH-16:
---------------------------------
> It might be better to fix the profiles issue first and then fix this one
+1. Also, wherever possible Crunch should use the same versions of libraries that Hadoop uses, to minimize version compatibility headaches.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Priority: Critical
> Labels: maven
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438620#comment-13438620 ]
Gabriel Reid commented on CRUNCH-16:
------------------------------------
Josh, about the various Avro dependencies, what do you think of just having some kind of static field somewhere that defines whether or not the capability of the ReflectDatumReader to properly handle all datum types present? This could then be set up in a static initializer, and then the creation of PTypes that combine Specific and Reflect could be disallowed at runtime (and the same logic would be needed to decide on whether certain unit tests should be run).
If that sounds like it's in line with what you were thinking, I can try to put that together later today and update the patch.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch, CRUNCH-16-avro-preparation.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417818#comment-13417818 ]
Josh Wills commented on CRUNCH-16:
----------------------------------
I will defer to your judgement on this one, Matthias.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Priority: Critical
> Labels: maven
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CRUNCH-16) Fix dependency versions
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthias Friedrich reassigned CRUNCH-16:
----------------------------------------
Assignee: Matthias Friedrich
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417899#comment-13417899 ]
Tom White commented on CRUNCH-16:
---------------------------------
Matthias - that would work too. The important point really is that it's not just the version of Hadoop that will be driven by a profile, but potentially some other libraries too (if Hadoop depends on different versions of them).
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Priority: Critical
> Labels: maven
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436582#comment-13436582 ]
Matthias Friedrich commented on CRUNCH-16:
------------------------------------------
Yes, me too, but running on top of Hadoop there's only so much we can do. Hadoop is a cool, robust distributed system but it just plain sucks as an execution environment. Java EE shows how to do it correctly, but sadly, these things don't seem to have priority.
So, what are we going to do?
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437261#comment-13437261 ]
Josh Wills commented on CRUNCH-16:
----------------------------------
I'm taking a crack at a patch for what is needed to have things work as seamlessly as possible from Avro 1.5.4 through Avro 1.7.1. My preference would be for the most recent version of Avro that has API compatibility w/Avro 1.5.3. It's a bit of a slog, but I think I'll get it done tomorrow.
I looked at the newest Crunch-16, and everything looks fine except for the versions on the protobuf dependencies-- I think 2.3.1, 2.4.0a, and 2.4.1 are all getting referenced at some point.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437569#comment-13437569 ]
Gabriel Reid commented on CRUNCH-16:
------------------------------------
I'll definitely take a look at this, but unfortunately it might be a couple of days before I can get to it.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch, CRUNCH-16-avro-preparation.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-16) Fix dependency versions
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated CRUNCH-16:
-----------------------------
Attachment: CRUNCH-16-avro-preparation.patch
Re: protobufs, cool, thanks.
The attached patch will pass all unit/integration tests w/Avro 1.5.4, 1.6.2, and 1.7.0, w/both Hadoop 1 and 2. The one thing I'm missing at this point is code to specifically disallow AvroTypes that combine Specific and Reflect-based schemas in the same MR job, since that will still fail in 1.5.4 and 1.6.2 but not in 1.7.0. We should have Gabriel and Christian take a look at this one before we integrate it.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch, CRUNCH-16-avro-preparation.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418060#comment-13418060 ]
Matthias Friedrich commented on CRUNCH-16:
------------------------------------------
@Rahul: I can think of four basic options here:
1. Drop support for CDH-3 (CDH-4 ships jackson 1.8.8)
2. Drop support for jackson-smile until CDH-3 isn't supported anymore
3. Ship recent versions of jackson-* and ask CDH-3 users to set HADOOP_USER_CLASSPATH_FIRST
4. Try if a version of jackson-smile works with the jackson-core/mapper-1.5.2
I like none of them. Option 1 may work because people who are conservative enough to still run CDH-3 (like us) probably won't use something as young as Crunch. Option 3 is one more way for users to shoot themselves in the foot. Option 4 doesn't work because the libraries are really incompatible (I tried several using our PageRankIT test case).
To be honest, I prefer option 2. People who really need smile can write those 50 LoC themselves, add the dependency, and set HADOOP_USER_CLASSPATH_FIRST.
Any ideas? Am I missing something?
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Priority: Critical
> Labels: maven
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417051#comment-13417051 ]
Matthias Friedrich commented on CRUNCH-16:
------------------------------------------
Re 2.: Crunch used to depend on the bundled Guava but was changed to a separate, real Guava in https://git-wip-us.apache.org/repos/asf?p=incubator-crunch.git;a=commit;h=9367826ee5516439852006c333702bd383241335. I think we should keep it this way, it's the best solution.
Long term I'm interested in reducing the number of dependencies Crunch requires. More dependencies mean more problems for users because of incompatibilities with application code. Guava is great in this regard because it replaces a lot of stuff from commons-*. In fact, we could trivially remove the dependency on commons-lang, but that's a different matter.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Priority: Critical
> Labels: maven
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CRUNCH-16) Fix dependency versions
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills resolved CRUNCH-16.
------------------------------
Resolution: Fixed
Fix Version/s: 0.3.0
Committed this, sans the avro 1.5.x versioning. I'll send an email to the dev list throwing out options on how to handle it.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Fix For: 0.3.0
>
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch, CRUNCH-16-avro-preparation.patch, CRUNCH-16-avro-reflect-specific-dynamic-init.patch.txt, CRUNCH-16-avro-reflect-specific-static-init.patch, CRUNCH-16-uber.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-16) Fix dependency versions
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated CRUNCH-16:
-----------------------------
Attachment: CRUNCH-16-avro-reflect-specific-static-init.patch
Gabriel, I had the exact thought that you did, and updated the patch w/just such an initializer. If you try to create an AvroType that combines reflect and specific schemas, we check the initializer and then throw a fail fast exception so that your job won't fail on the cluster. Take another look?
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch, CRUNCH-16-avro-preparation.patch, CRUNCH-16-avro-reflect-specific-static-init.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438782#comment-13438782 ]
Gabriel Reid commented on CRUNCH-16:
------------------------------------
Josh, that's almost exactly what I was talking about, but I was actually thinking of having the CAN_COMBINE_SPECIFIC_AND_REFLECT_SCHEMAS automatically initialized (in a static initializer block) by a kind of mini-test that would actually actually attempt to combine both in a single schema and set the static field as a result of that test. However, I'm kind of on the fence as to whether it's worth it adding those kinds of moving parts to satisfy a use case that probably won't come up much.
I'm ok with it either way, but I don't mind taking a crack at implementing that idea if you think it's worth including it.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Assignee: Matthias Friedrich
> Priority: Critical
> Labels: maven
> Attachments: 0001-CRUNCH-16-Fix-dependency-versions.patch, CRUNCH-16-avro-preparation.patch, CRUNCH-16-avro-reflect-specific-static-init.patch
>
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-16) Fix dependency versions
Posted by "Rahul Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418135#comment-13418135 ]
Rahul Sharma commented on CRUNCH-16:
------------------------------------
I can think of one more way of solving this. This dependency is being used in PTYPEs class. We can refactor the piece that uses this code to a separate class.
There will be one more implementation of this piece without the dependency. Both these will implementations will reside in separate src folders like src/main/CDH3/java and src/main/CHD4above/java. The src folders will be enabled on the profile that gets built. Hadoop1 will have this code and cdh3 version will not.
> Fix dependency versions
> -----------------------
>
> Key: CRUNCH-16
> URL: https://issues.apache.org/jira/browse/CRUNCH-16
> Project: Crunch
> Issue Type: Task
> Reporter: Matthias Friedrich
> Priority: Critical
> Labels: maven
>
> Right now, the compile time classpath doesn't exactly match the runtime classpath. For example, Hadoop CDH3u4 comes with slf4j-api-1.4.3 while Crunch references slf4j-api-1.6.3. This could result in NoSuchMethodError and other nasty problems.
> We have to make sure that version numbers match (check "mvn dependency:tree") and that we don't ship artifacts that are already present on the runtime classpath. If our dependencies introduce incompatible transitive dependencies, we might have to downgrade them.
> Note: Maven's dependency resolution is order-dependent, see http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira