You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Niels Basjes (JIRA)" <ji...@apache.org> on 2016/05/11 11:59:12 UTC
[jira] [Commented] (PIG-4111) Make Pig compiles with avro-1.7.7
[ https://issues.apache.org/jira/browse/PIG-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280016#comment-15280016 ]
Niels Basjes commented on PIG-4111:
-----------------------------------
Quick response:
# I traced this Avro change back to https://issues.apache.org/jira/browse/AVRO-1373 which was released as part of Avro 1.7.6. By the looks of it the code went from a "Hardcoded magic number '1' specific for the AvroOutputFormat" to "Let's use the centrally defined value in de CodecFactory that is based on the values defined in the (jdk) Deflater class". To me this looks like "Ok to change".
# I haven't tried this but I think it is safe to say people should include the 'latest' version of various libraries in their project. So if someone chooses to upgrade to a newer version of Pig then should also upgrade the versions of other libraries. So as far as I can tell this would only affect people with a custom UDF that has an 'old' version of AVRO builtin.
*How about updating to Avro 1.8.0?*
I did a quick check if I could update the patch to the latest version of Avro (1.8.0) yet that yields this error (I used the docker image to build).
{code}
BUILD FAILED
/home/nbasjes/pig/build.xml:1704: impossible to ivy retrieve: java.lang.RuntimeException: problem during retrieve of org.apache.pig#pig: java.lang.RuntimeException: Multiple artifacts of the module org.apache.avro#trevni-core;1.8.0 are retrieved to the same file! Update the retrieve pattern to fix this error.
{code}
What I found is that this does not occur with Avro 1.7.7
I traced it to the fact that the trevni-core-1.8.0-tests.jar is somehow erroneously downloaded as both a tests-jar AND according to ivy the same file is also a 'normal jar'.
In ~/.ivy2/cache/org.apache.avro/trevni-core/ivydata-1.8.0.properties I see this:
{quote}
#ivy cached data file for org.apache.avro#trevni-core;1.8.0
#Wed May 11 11:37:07 UTC 2016
*artifact\:trevni-core\#jar\#jar\#-236753692.location=http\://repo2.maven.org/maven2/org/apache/avro/trevni-core/1.8.0/trevni-core-1.8.0-tests.jar*
artifact\:trevni-core\#jar\#jar\#-236753692.is-local=false
resolver=maven2
artifact\:ivy\#ivy\#xml\#-1753783300.is-local=false
artifact\:trevni-core\#jar\#jar\#901594073.location=http\://repo2.maven.org/maven2/org/apache/avro/trevni-core/1.8.0/trevni-core-1.8.0.jar
artifact\:ivy\#ivy\#xml\#-1753783300.location=http\://repo2.maven.org/maven2/org/apache/avro/trevni-core/1.8.0/trevni-core-1.8.0.pom
artifact\:trevni-core\#pom.original\#pom\#-1740810561.is-local=false
artifact\:trevni-core\#pom.original\#pom\#-1740810561.location=http\://repo2.maven.org/maven2/org/apache/avro/trevni-core/1.8.0/trevni-core-1.8.0.pom
artifact\:trevni-core\#jar\#jar\#901594073.is-local=false
*artifact\:trevni-core\#test-jar\#jar\#38927841.location=http\://repo2.maven.org/maven2/org/apache/avro/trevni-core/1.8.0/trevni-core-1.8.0-tests.jar*
artifact.resolver=default
artifact\:trevni-core\#test-jar\#jar\#38927841.is-local=false
{quote}
I checked the upstream Maven repos and I have not been able to find a difference that may explain this.
I'm quite willing to dive into this but right now I'm stuck.
[~rohini] can you assist here?
> Make Pig compiles with avro-1.7.7
> ---------------------------------
>
> Key: PIG-4111
> URL: https://issues.apache.org/jira/browse/PIG-4111
> Project: Pig
> Issue Type: Improvement
> Components: build
> Affects Versions: 0.13.0
> Reporter: fang fang chen
> Assignee: fang fang chen
> Fix For: site
>
> Attachments: PIG-4111-1.patch, PIG-4111.patch
>
>
> pig-0.13 compile failed with avro-1.7.7:
> [javac] /root/ff/git/pig/src/org/apache/pig/impl/util/avro/AvroRecordWriter.java:23: error: cannot find symbol
> [javac] import static org.apache.avro.mapred.AvroOutputFormat.DEFAULT_DEFLATE_LEVEL;
> [javac] ^
> [javac] symbol: static DEFAULT_DEFLATE_LEVEL
> [javac] location: class
> [javac] where T is a type-variable:
> [javac] T extends Object declared in class AvroOutputFormat
> [javac] /root/ff/git/pig/src/org/apache/pig/impl/util/avro/AvroRecordWriter.java:76: error: cannot find symbol
> [javac] DEFAULT_DEFLATE_LEVEL);
> [javac] ^
> [javac] symbol: variable DEFAULT_DEFLATE_LEVEL
> [javac] location: class AvroRecordWriter
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)