You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Andrew Ash (JIRA)" <ji...@apache.org> on 2014/02/10 08:33:21 UTC
[jira] [Commented] (AVRO-1453) Release version of avro-tools
compiled against hadoop2
[ https://issues.apache.org/jira/browse/AVRO-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13896287#comment-13896287 ]
Andrew Ash commented on AVRO-1453:
----------------------------------
I get a hadoop2.jar with that patch applied, but for some reason it's forgotten about the hdfs:// filesystem scheme. I think maybe the shading doesn't work properly with the hadoop2 classifier?
After building with -Dhadoop.version=2
{noformat}
$ find . -name '*jar'
./target/avro-tools-1.7.6-hadoop2.jar
./target/avro-tools-1.7.6-nodeps.jar
./target/avro-tools-1.7.6.jar
$ java -jar ./target/avro-tools-1.7.6.jar concat <hdfs path1> <hdfs path2> /tmp/output
no main manifest attribute, in ./target/avro-tools-1.7.6.jar
$ java -jar ./target/avro-tools-1.7.6-hadoop2.jar concat <hdfs path1> <hdfs path2> /tmp/output
Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2130)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2137)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2176)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2158)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
at org.apache.avro.tool.Util.openFromFS(Util.java:88)
at org.apache.avro.tool.Util.fileOrStdin(Util.java:60)
at org.apache.avro.tool.ConcatTool.run(ConcatTool.java:69)
at org.apache.avro.tool.Main.run(Main.java:84)
at org.apache.avro.tool.Main.main(Main.java:73)
{noformat}
After building without -Dhadoop.version=2 parameter:
{noformat}
$ find . -name '*jar'
./target/avro-tools-1.7.6-nodeps.jar
./target/avro-tools-1.7.6.jar
./target/original-avro-tools-1.7.6.jar
$ java -jar ./target/avro-tools-1.7.6.jar concat <hdfs path1> <hdfs path2> /tmp/output
Exception in thread "main" org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4
at org.apache.hadoop.ipc.Client.call(Client.java:1066)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
at org.apache.avro.tool.Util.openFromFS(Util.java:88)
at org.apache.avro.tool.Util.fileOrStdin(Util.java:60)
at org.apache.avro.tool.ConcatTool.run(ConcatTool.java:69)
at org.apache.avro.tool.Main.run(Main.java:84)
at org.apache.avro.tool.Main.main(Main.java:73)
{noformat}
> Release version of avro-tools compiled against hadoop2
> ------------------------------------------------------
>
> Key: AVRO-1453
> URL: https://issues.apache.org/jira/browse/AVRO-1453
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.7.6
> Environment: CDH4.4.0
> Reporter: Andrew Ash
> Assignee: Doug Cutting
> Fix For: 1.7.7
>
> Attachments: AVRO-1453.patch
>
>
> When attempting to use avro-tools-1.7.6.jar against my CDH4.4.0 cluster I get this error:
> {code}
> Feb 06, 2014 9:57:13 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
> WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Exception in thread "main" org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4
> at org.apache.hadoop.ipc.Client.call(Client.java:1066)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
> at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
> at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
> at org.apache.avro.tool.Util.openFromFS(Util.java:88)
> at org.apache.avro.tool.Util.fileOrStdin(Util.java:60)
> at org.apache.avro.tool.ConcatTool.run(ConcatTool.java:69)
> at org.apache.avro.tool.Main.run(Main.java:84)
> at org.apache.avro.tool.Main.main(Main.java:73)
> {code}
> I believe that the version released on maven is compiled against hadoop 1, and I'm running hadoop 2. Can we release a version that's compiled against hadoop2?
> https://search.maven.org/#artifactdetails%7Corg.apache.avro%7Cavro-tools%7C1.7.6%7Cjar
> Thanks!
> Andrew
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)