[jira] [Commented] (SQOOP-1359) Fix avro versions in Sqoop to stop shipping hadoop1 jars with hadoop2

["Venkat Ranganathan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SQOOP-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14068554#comment-14068554 ] 

Venkat Ranganathan commented on SQOOP-1359:
-------------------------------------------

This is getting tricky from the build point of view.  Some projects do not use hadoop2/hadoop1 classifier (and are not dependent on avro-mapred.jar hadoop1/hadoop2 specifiic classes).   So it is more likely that you have in your ivy and m2 cache, the avro-mapred.jar cached.    I added the classifier for hadoop2 builds, but the build fails when the cached artifact in the local ivy and m2 caches do not have the right classifier.   We have to clean the avro-mapred cached jars in m2 and ivy caches to make the sqoop build successful and this may affect building other components.  I tried to update ivy to the latest release version 2.3.0 but still ivy does not download the jar with hadoop2 classifier if it finds something without the classifier in the m2 cache.

Looking at Sqoop, also we don't really depend on avro-mapred hadoop1/hadoop2 dependent classes (those that deal with TaskAttamptContexts).   

I think we have two options:
Just update avro version alone without specifying classifiers (which avro version would be question - 1.7.4 is used by hadoop, and 1.7.5 is used by hive and they are some incompatibilities between the versions).  

Or recommend users to nuke the avro-mapred cached jars before building sqoop and any other avro-mapred dependent builds.

Thoughts?

> Fix avro versions in Sqoop to stop shipping hadoop1  jars with hadoop2
> ----------------------------------------------------------------------
>
>                 Key: SQOOP-1359
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1359
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.4.4, 1.4.5
>            Reporter: Venkat Ranganathan
>            Assignee: Venkat Ranganathan
>             Fix For: 1.4.5
>
>
> AVRO-1170 fixed avro-mapred for hadoop2 and we now have avro-mapred hadoop1 and hadoop2 specific jars.   Our dependency should be updated to use the right jars.   I recently found similar issues with Hive shipping avro-mapred hadoop1 jars (HIVE-7240) causing issues in certain integration scenarios.



--
This message was sent by Atlassian JIRA
(v6.2#6252)