You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2011/09/23 09:18:26 UTC

[jira] [Commented] (PIG-2262) AvroStorage dependencies are missing from the release tarball

    [ https://issues.apache.org/jira/browse/PIG-2262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113208#comment-13113208 ] 

Daniel Dai commented on PIG-2262:
---------------------------------

There are a couple issues with this approach, actually most of issues are not specific to AvroStorage, it is how we deal with UDF dependent jars:

1. Pig don't automatically ship all classes in pig-withouthadoop.jar
   We also need to make code change in JarManager.jar to denote the package to ship. Putting a jar into pig-withouthadoop.jar alone is equal to put this jar in classpath. This mechanism confusing and we shall stop putting more jars into pig-withouthadoop.jar

2. Conflict with hadoop bundled jars
   Hadoop 20.204 bundles jackson-1.0.1, which is too old for AvroLoader. In frontend, we can force hadoop take our jackson-1.7.3 by setting flag HADOOP_USER_CLASSPATH_FIRST=true. But in the backend, seems hadoop always pick bundled jackson-1.0.1, which results a job failure.

3. Do we need to bundle piggybank dependent jars?
   We don't even bundle hbase.jar though HbaseLoader is in builtin. Further, these jars are not even in Pig distribution. They are ivy dependencies and will only be retrieved during compilation. My thinking is we need to bundle some popular jars (hbase.jar, avro.jar, etc) in lib so user knows where to find it when needed. But we don't want to ship all those jars to the backend. Ideally Pig should be smart enough to ship jars when needed (as we do for jython.jar)

> AvroStorage dependencies are missing from the release tarball
> -------------------------------------------------------------
>
>                 Key: PIG-2262
>                 URL: https://issues.apache.org/jira/browse/PIG-2262
>             Project: Pig
>          Issue Type: Bug
>          Components: build, piggybank
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: PIG-2262.patch
>
>
> This makes AvroStorage hard to use, since users have to download the dependencies manually, or build Pig themselves.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira