You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by ant burton <ap...@gmail.com> on 2017/09/05 21:52:00 UTC
Flink on AWS EMR Protobuf
Hello,
Has anybody experienced the following error on AWS EMR 5.8.0 with Flink 1.3.1
java.lang.ClassCastException: org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto cannot be cast to com.google.protobuf.Message
Thanks,
Re: Flink on AWS EMR Protobuf
Posted by Timo Walther <tw...@apache.org>.
I'm not sure if this is a Flink issue. It popped up on other non-Flink
projects as well:
http://community.cloudera.com/t5/Storage-Random-Access-HDFS/map-red-over-hbase-in-cdh-5-7/td-p/43902
I would defenitely check your dependencies. This looks like conflicting
versions in your classpaths.
Timo
Am 07.09.17 um 11:28 schrieb ant burton:
> Please find a stack trace below, and thank you for taking a look :-)
>
> java.lang.ClassCastException:
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto
> cannot be cast to com.google.protobuf.Message
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:226)
> at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache.flink.yarn.Utils.registerLocalResource(Utils.java:150)
> at
> org.apache.flink.yarn.Utils.createTaskExecutorContext(Utils.java:383)
> at
> org.apache.flink.yarn.YarnApplicationMasterRunner.runApplicationMaster(YarnApplicationMasterRunner.java:324)
> at
> org.apache.flink.yarn.YarnApplicationMasterRunner$1.call(YarnApplicationMasterRunner.java:195)
> at
> org.apache.flink.yarn.YarnApplicationMasterRunner$1.call(YarnApplicationMasterRunner.java:192)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
> at
> org.apache.flink.yarn.YarnApplicationMasterRunner.run(YarnApplicationMasterRunner.java:192)
> at
> org.apache.flink.yarn.YarnApplicationMasterRunner.main(YarnApplicationMasterRunner.java:116)
> 2017-09-07 08:29:31,475 INFO
> akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down
> remote daemon.
> 2017-09-07 08:29:31,478 INFO
> akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote daemon
> shut down; proceeding with flushing remote transports.
> End of LogType:jobmanager.log
>
>
>> On 6 Sep 2017, at 13:36, Aljoscha Krettek <aljoscha@apache.org
>> <ma...@apache.org>> wrote:
>>
>> Hi,
>>
>> Could you please give a bit more context around that exception? Maybe
>> a log or a full stack trace.
>>
>> Best,
>> Aljoscha
>>
>>> On 5. Sep 2017, at 23:52, ant burton <apburton84@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> Hello,
>>>
>>> Has anybody experienced the following error on AWS EMR 5.8.0 with
>>> Flink 1.3.1
>>>
>>> /java.lang.ClassCastException:
>>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto
>>> cannot be cast to com.google.protobuf.Message/
>>>
>>> Thanks,
>>>
>>>
>>
>
Re: Flink on AWS EMR Protobuf
Posted by ant burton <ap...@gmail.com>.
Please find a stack trace below, and thank you for taking a look :-)
java.lang.ClassCastException: org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto cannot be cast to com.google.protobuf.Message
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:226)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
at org.apache.flink.yarn.Utils.registerLocalResource(Utils.java:150)
at org.apache.flink.yarn.Utils.createTaskExecutorContext(Utils.java:383)
at org.apache.flink.yarn.YarnApplicationMasterRunner.runApplicationMaster(YarnApplicationMasterRunner.java:324)
at org.apache.flink.yarn.YarnApplicationMasterRunner$1.call(YarnApplicationMasterRunner.java:195)
at org.apache.flink.yarn.YarnApplicationMasterRunner$1.call(YarnApplicationMasterRunner.java:192)
at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.yarn.YarnApplicationMasterRunner.run(YarnApplicationMasterRunner.java:192)
at org.apache.flink.yarn.YarnApplicationMasterRunner.main(YarnApplicationMasterRunner.java:116)
2017-09-07 08:29:31,475 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon.
2017-09-07 08:29:31,478 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports.
End of LogType:jobmanager.log
> On 6 Sep 2017, at 13:36, Aljoscha Krettek <al...@apache.org> wrote:
>
> Hi,
>
> Could you please give a bit more context around that exception? Maybe a log or a full stack trace.
>
> Best,
> Aljoscha
>
>> On 5. Sep 2017, at 23:52, ant burton <apburton84@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hello,
>>
>> Has anybody experienced the following error on AWS EMR 5.8.0 with Flink 1.3.1
>>
>> java.lang.ClassCastException: org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto cannot be cast to com.google.protobuf.Message
>>
>> Thanks,
>>
>>
>
Re: Flink on AWS EMR Protobuf
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
Could you please give a bit more context around that exception? Maybe a log or a full stack trace.
Best,
Aljoscha
> On 5. Sep 2017, at 23:52, ant burton <ap...@gmail.com> wrote:
>
> Hello,
>
> Has anybody experienced the following error on AWS EMR 5.8.0 with Flink 1.3.1
>
> java.lang.ClassCastException: org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto cannot be cast to com.google.protobuf.Message
>
> Thanks,
>
>