You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by wesley dias <we...@outlook.com> on 2013/10/28 21:02:35 UTC

Exception in thread "SinkRunne​r-PollingR​unner-Defa​ultSinkPro​cessor" on writing to HDFS sink

Hi,
 
I am using the latest flume build 1.4 along with hadoop 2.2.0 to collect twitter stream and write to HDFS sink. However, on writing to HDFS I get the following error:
 
2013-10-28 09:41:23,481 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:422)] process failed
java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$SetOwnerRequestProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
     at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
     at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
     at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
     at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
    at java.lang.Class.privateGetPublicMethods(Class.java:2651)
    at java.lang.Class.privateGetPublicMethods(Class.java:2661)
     at java.lang.Class.getMethods(Class.java:1467)
 
 I am currently new to flume and any help shall be really appreciated.
 
Regards,
 
Wesley

Re: Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" on writing to HDFS sink

Posted by David Sinclair <ds...@chariotsolutions.com>.
Glad to hear that was the issue.

dave


On Tue, Oct 29, 2013 at 12:53 AM, wesley dias <we...@outlook.com>wrote:

> Thanks Roshan. But issue is now resolved. I compiled flume 1.4 from source
> using protobuf 2.5 and this solved the below mentioned issue. My guess is
> that the distribution available for flume 1.4 from the release site must
> have been compiled using protobuf 2.4
>
> Regards,
>
> Wesley
>
> ------------------------------
> Date: Mon, 28 Oct 2013 13:41:52 -0700
>
> Subject: Re: Exception in thread
> "SinkRunner-PollingRunner-DefaultSinkProcessor" on writing to HDFS sink
> From: roshan@hortonworks.com
> To: wesleydias@outlook.com
>
> HDP already includes hadoop 2.2.0 + Flume 1.4. You'd be better of just
> using instead of building and testing that combo out on your own.
>
> http://hortonworks.com/products/hdp-2/?b=2#overview
>
>
> On Mon, Oct 28, 2013 at 1:19 PM, wesley dias <we...@outlook.com>wrote:
>
> Hi David,
>
> Thanks for the quick reply.Does this mean that I would need to compile the
> flume from source using protobuf 2.5 instead of directly using the flume
> distribution available for download from the site? Is flume 1.4 based of
> protobuf 2.4?
> I am not sure of the same hence I am asking these questions.
>
> Regards,
>
> Wesley
> ------------------------------
> Date: Mon, 28 Oct 2013 16:11:16 -0400
> Subject: Re: Exception in thread
> "SinkRunner-PollingRunner-DefaultSinkProcessor" on writing to HDFS sink
> From: dsinclair@chariotsolutions.com
> To: user@flume.apache.org
>
>
> Looks like you have a protocol buffer version problem. Do you have a
> dependency on 2.5.0 version of protocol buffers in your pom?
>
> see
>
> http://code.google.com/p/protobuf/issues/detail?id=493
>
>
>
> On Mon, Oct 28, 2013 at 4:02 PM, wesley dias <we...@outlook.com>wrote:
>
> Hi,
>
> I am using the latest flume build 1.4 along with hadoop 2.2.0 to collect
> twitter stream and write to HDFS sink. However, on writing to HDFS I get
> the following error:
>
> 2013-10-28 09:41:23,481 (SinkRunner-PollingRunner-**DefaultSinkProcessor)
> [ERROR - org.apache.flume.sink.hdfs.**HDFSEventSink.process(**HDFSEventSink.java:422)]
> process failed
> java.lang.VerifyError: class org.apache.hadoop.hdfs.**protocol.proto.**
> ClientNamenodeProtocolProtos$**SetOwnerRequestProto overrides final
> method getUnknownFields.()Lcom/**google/protobuf/**UnknownFieldSet;
>     at java.lang.ClassLoader.**defineClass1(Native Method)
>     at java.lang.ClassLoader.**defineClass(ClassLoader.java:**800)
>     at java.security.**SecureClassLoader.defineClass(**
> SecureClassLoader.java:142)
>     at java.net.URLClassLoader.**defineClass(URLClassLoader.**java:449)
>     at java.net.URLClassLoader.**access$100(URLClassLoader.**java:71)
>     at java.net.URLClassLoader$1.run(**URLClassLoader.java:361)
>     at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
>     at java.security.**AccessController.doPrivileged(**Native Method)
>     at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
>     at java.lang.ClassLoader.**loadClass(ClassLoader.java:**425)
>     at sun.misc.Launcher$**AppClassLoader.loadClass(**Launcher.java:308)
>     at java.lang.ClassLoader.**loadClass(ClassLoader.java:**358)
>     at java.lang.Class.**getDeclaredMethods0(Native Method)
>     at java.lang.Class.**privateGetDeclaredMethods(**Class.java:2531)
>     at java.lang.Class.**privateGetPublicMethods(Class.**java:2651)
>     at java.lang.Class.**privateGetPublicMethods(Class.**java:2661)
>     at java.lang.Class.getMethods(**Class.java:1467)
>
>  I am currently new to flume and any help shall be really appreciated.
>
> Regards,
>
> Wesley
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

RE: Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" on writing to HDFS sink

Posted by wesley dias <we...@outlook.com>.
Thanks Roshan. But issue is now resolved. I compiled flume 1.4 from source using protobuf 2.5 and this solved the below mentioned issue. My guess is that the distribution available for flume 1.4 from the release site must have been compiled using protobuf 2.4
 
Regards,
 
Wesley
 
Date: Mon, 28 Oct 2013 13:41:52 -0700
Subject: Re: Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" on writing to HDFS sink
From: roshan@hortonworks.com
To: wesleydias@outlook.com

HDP already includes hadoop 2.2.0 + Flume 1.4. You'd be better of just using instead of building and testing that combo out on your own.
http://hortonworks.com/products/hdp-2/?b=2#overview



On Mon, Oct 28, 2013 at 1:19 PM, wesley dias <we...@outlook.com> wrote:




Hi David,
 
Thanks for the quick reply.Does this mean that I would need to compile the flume from source using protobuf 2.5 instead of directly using the flume distribution available for download from the site? Is flume 1.4 based of protobuf 2.4?

I am not sure of the same hence I am asking these questions.
 
Regards,
 
Wesley 
Date: Mon, 28 Oct 2013 16:11:16 -0400
Subject: Re: Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" on writing to HDFS sink

From: dsinclair@chariotsolutions.com
To: user@flume.apache.org


Looks like you have a protocol buffer version problem. Do you have a dependency on 2.5.0 version of protocol buffers in your pom?
see 
http://code.google.com/p/protobuf/issues/detail?id=493





On Mon, Oct 28, 2013 at 4:02 PM, wesley dias <we...@outlook.com> wrote:




Hi,
 
I am using the latest flume build 1.4 along with hadoop 2.2.0 to collect twitter stream and write to HDFS sink. However, on writing to HDFS I get the following error:
 
2013-10-28 09:41:23,481 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:422)] process failed


java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$SetOwnerRequestProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;


     at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)


    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
     at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)


    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
     at java.net.URLClassLoader.findClass(URLClassLoader.java:354)


    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)


     at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
    at java.lang.Class.privateGetPublicMethods(Class.java:2651)


    at java.lang.Class.privateGetPublicMethods(Class.java:2661)
     at java.lang.Class.getMethods(Class.java:1467)
 
 I am currently new to flume and any help shall be really appreciated.


 
Regards,
 
Wesley
 		 	   		  

 		 	   		  





CONFIDENTIALITY NOTICENOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

RE: Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" on writing to HDFS sink

Posted by wesley dias <we...@outlook.com>.
Hi David,
 
Thanks for the quick reply.Does this mean that I would need to compile the flume from source using protobuf 2.5 instead of directly using the flume distribution available for download from the site? Is flume 1.4 based of protobuf 2.4?
I am not sure of the same hence I am asking these questions.
 
Regards,
 
Wesley 
Date: Mon, 28 Oct 2013 16:11:16 -0400
Subject: Re: Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" on writing to HDFS sink
From: dsinclair@chariotsolutions.com
To: user@flume.apache.org

Looks like you have a protocol buffer version problem. Do you have a dependency on 2.5.0 version of protocol buffers in your pom?
see 
http://code.google.com/p/protobuf/issues/detail?id=493




On Mon, Oct 28, 2013 at 4:02 PM, wesley dias <we...@outlook.com> wrote:




Hi,
 
I am using the latest flume build 1.4 along with hadoop 2.2.0 to collect twitter stream and write to HDFS sink. However, on writing to HDFS I get the following error:
 
2013-10-28 09:41:23,481 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:422)] process failed

java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$SetOwnerRequestProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;

     at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
     at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)

    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
     at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

     at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
    at java.lang.Class.privateGetPublicMethods(Class.java:2651)

    at java.lang.Class.privateGetPublicMethods(Class.java:2661)
     at java.lang.Class.getMethods(Class.java:1467)
 
 I am currently new to flume and any help shall be really appreciated.

 
Regards,
 
Wesley

Re: Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" on writing to HDFS sink

Posted by David Sinclair <ds...@chariotsolutions.com>.
Looks like you have a protocol buffer version problem. Do you have a
dependency on 2.5.0 version of protocol buffers in your pom?

see

http://code.google.com/p/protobuf/issues/detail?id=493



On Mon, Oct 28, 2013 at 4:02 PM, wesley dias <we...@outlook.com> wrote:

> Hi,
>
> I am using the latest flume build 1.4 along with hadoop 2.2.0 to collect
> twitter stream and write to HDFS sink. However, on writing to HDFS I get
> the following error:
>
> 2013-10-28 09:41:23,481 (SinkRunner-PollingRunner-**DefaultSinkProcessor)
> [ERROR - org.apache.flume.sink.hdfs.**HDFSEventSink.process(**HDFSEventSink.java:422)]
> process failed
> java.lang.VerifyError: class org.apache.hadoop.hdfs.**protocol.proto.**
> ClientNamenodeProtocolProtos$**SetOwnerRequestProto overrides final
> method getUnknownFields.()Lcom/**google/protobuf/**UnknownFieldSet;
>     at java.lang.ClassLoader.**defineClass1(Native Method)
>     at java.lang.ClassLoader.**defineClass(ClassLoader.java:**800)
>     at java.security.**SecureClassLoader.defineClass(**
> SecureClassLoader.java:142)
>     at java.net.URLClassLoader.**defineClass(URLClassLoader.**java:449)
>     at java.net.URLClassLoader.**access$100(URLClassLoader.**java:71)
>     at java.net.URLClassLoader$1.run(**URLClassLoader.java:361)
>     at java.net.URLClassLoader$1.run(**URLClassLoader.java:355)
>     at java.security.**AccessController.doPrivileged(**Native Method)
>     at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354)
>     at java.lang.ClassLoader.**loadClass(ClassLoader.java:**425)
>     at sun.misc.Launcher$**AppClassLoader.loadClass(**Launcher.java:308)
>     at java.lang.ClassLoader.**loadClass(ClassLoader.java:**358)
>     at java.lang.Class.**getDeclaredMethods0(Native Method)
>     at java.lang.Class.**privateGetDeclaredMethods(**Class.java:2531)
>     at java.lang.Class.**privateGetPublicMethods(Class.**java:2651)
>     at java.lang.Class.**privateGetPublicMethods(Class.**java:2661)
>     at java.lang.Class.getMethods(**Class.java:1467)
>
>  I am currently new to flume and any help shall be really appreciated.
>
> Regards,
>
> Wesley
>