You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by shekhar sharma <sh...@gmail.com> on 2012/06/20 19:48:36 UTC
Unable to transport the event to Avro Source
Hello,
I am encountering an exception when i am trying to transport the events to
flume avro source.
Basically i have created an agent consisting of avro source and my custom
sink.
Previously i was using exec source, but due to its limitations i am not
using it anymore.
Since Avro Source is listening on a port, so i have created a simple
application, which opens a socket to that port and send the events through
this socket.
Socket connection is successful, but when i send the events i got the
following error:
2012-06-20 10:34:15,767 INFO source.AvroSource: Avro source
starting:AvroSource: { bindAddress:localhost port:41414 }
2012-06-20 10:35:52,588 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
127.0.0.1:46240 => /127.0.0.1:41414] OPEN
2012-06-20 10:35:52,612 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
127.0.0.1:46240 => /127.0.0.1:41414] BOUND: /127.0.0.1:41414
2012-06-20 10:35:52,612 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
127.0.0.1:46240 => /127.0.0.1:41414] CONNECTED: /127.0.0.1:46240
2012-06-20 10:35:52,792 WARN ipc.NettyServer: Unexpected exception from
downstream.
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.<init>(ArrayList.java:112)
at
org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:154)
at
org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:131)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Is there something else i need to perform?
Regards,
Som Shekhar Sharma
Re: Unable to transport the event to Avro Source
Posted by shekhar sharma <sh...@gmail.com>.
Thanks Will and Hari for your valuable guidance. Now i am able to transport
the events from my external program to the Avro Source.Its pretty simple
with 4 lines of code.
Regards,
Som Shekhar
On Wed, Jun 20, 2012 at 11:35 PM, shekhar sharma <sh...@gmail.com>wrote:
> Thanks Will.. i am looking into it...
>
> Regards,
> Som
>
>
> On Wed, Jun 20, 2012 at 11:32 PM, shekhar sharma <sh...@gmail.com>wrote:
>
>> Hari, can you suggest me what needs to be done to convert the events into
>> Avro format. I am also trying to find out, but it would be great if you can
>> give me some pointers to start.
>>
>> Regards,
>> Som
>>
>>
>> On Wed, Jun 20, 2012 at 11:30 PM, shekhar sharma <sh...@gmail.com>wrote:
>>
>>> Thanks hari for prompt reply. I dont have a sink that writes to avro
>>> source.
>>> Actually i am writing a external program which will send the events to
>>> the Avro Source. you mean to say, my external program should convert the
>>> events into avro format and then i can send those event through a socket.
>>>
>>> Regards,
>>> SOm
>>>
>>>
>>> On Wed, Jun 20, 2012 at 11:22 PM, Hari Shreedharan <
>>> hshreedharan@cloudera.com> wrote:
>>>
>>>> Hi Som,
>>>>
>>>> AvroSource is designed to received only Avro formatted events, so if
>>>> you write arbitrary data to it, the AvroSource will behave in an undefined
>>>> way. If you have a sink that writes to the AvroSource, it should be using
>>>> the Flume SDK to send the event in the Flume Avro format. If you want to
>>>> write arbitrary data, I suggest taking a look at the netcat source.
>>>>
>>>> Thanks
>>>> Hari
>>>>
>>>> --
>>>> Hari Shreedharan
>>>>
>>>> On Wednesday, June 20, 2012 at 10:48 AM, shekhar sharma wrote:
>>>>
>>>> Hello,
>>>> I am encountering an exception when i am trying to transport the events
>>>> to flume avro source.
>>>> Basically i have created an agent consisting of avro source and my
>>>> custom sink.
>>>> Previously i was using exec source, but due to its limitations i am not
>>>> using it anymore.
>>>>
>>>> Since Avro Source is listening on a port, so i have created a simple
>>>> application, which opens a socket to that port and send the events through
>>>> this socket.
>>>> Socket connection is successful, but when i send the events i got the
>>>> following error:
>>>>
>>>> 2012-06-20 10:34:15,767 INFO source.AvroSource: Avro source
>>>> starting:AvroSource: { bindAddress:localhost port:41414 }
>>>> 2012-06-20 10:35:52,588 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
>>>> 127.0.0.1:46240 => /127.0.0.1:41414] OPEN
>>>> 2012-06-20 10:35:52,612 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
>>>> 127.0.0.1:46240 => /127.0.0.1:41414] BOUND: /127.0.0.1:41414
>>>> 2012-06-20 10:35:52,612 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
>>>> 127.0.0.1:46240 => /127.0.0.1:41414] CONNECTED: /127.0.0.1:46240
>>>> 2012-06-20 10:35:52,792 WARN ipc.NettyServer: Unexpected exception from
>>>> downstream.
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> at java.util.ArrayList.<init>(ArrayList.java:112)
>>>> at
>>>> org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:154)
>>>> at
>>>> org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:131)
>>>> at
>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
>>>> at
>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
>>>> at
>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
>>>> at
>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> Is there something else i need to perform?
>>>>
>>>> Regards,
>>>> Som Shekhar Sharma
>>>>
>>>>
>>>>
>>>
>>
>
Re: Unable to transport the event to Avro Source
Posted by shekhar sharma <sh...@gmail.com>.
Thanks Will.. i am looking into it...
Regards,
Som
On Wed, Jun 20, 2012 at 11:32 PM, shekhar sharma <sh...@gmail.com>wrote:
> Hari, can you suggest me what needs to be done to convert the events into
> Avro format. I am also trying to find out, but it would be great if you can
> give me some pointers to start.
>
> Regards,
> Som
>
>
> On Wed, Jun 20, 2012 at 11:30 PM, shekhar sharma <sh...@gmail.com>wrote:
>
>> Thanks hari for prompt reply. I dont have a sink that writes to avro
>> source.
>> Actually i am writing a external program which will send the events to
>> the Avro Source. you mean to say, my external program should convert the
>> events into avro format and then i can send those event through a socket.
>>
>> Regards,
>> SOm
>>
>>
>> On Wed, Jun 20, 2012 at 11:22 PM, Hari Shreedharan <
>> hshreedharan@cloudera.com> wrote:
>>
>>> Hi Som,
>>>
>>> AvroSource is designed to received only Avro formatted events, so if you
>>> write arbitrary data to it, the AvroSource will behave in an undefined way.
>>> If you have a sink that writes to the AvroSource, it should be using the
>>> Flume SDK to send the event in the Flume Avro format. If you want to write
>>> arbitrary data, I suggest taking a look at the netcat source.
>>>
>>> Thanks
>>> Hari
>>>
>>> --
>>> Hari Shreedharan
>>>
>>> On Wednesday, June 20, 2012 at 10:48 AM, shekhar sharma wrote:
>>>
>>> Hello,
>>> I am encountering an exception when i am trying to transport the events
>>> to flume avro source.
>>> Basically i have created an agent consisting of avro source and my
>>> custom sink.
>>> Previously i was using exec source, but due to its limitations i am not
>>> using it anymore.
>>>
>>> Since Avro Source is listening on a port, so i have created a simple
>>> application, which opens a socket to that port and send the events through
>>> this socket.
>>> Socket connection is successful, but when i send the events i got the
>>> following error:
>>>
>>> 2012-06-20 10:34:15,767 INFO source.AvroSource: Avro source
>>> starting:AvroSource: { bindAddress:localhost port:41414 }
>>> 2012-06-20 10:35:52,588 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
>>> 127.0.0.1:46240 => /127.0.0.1:41414] OPEN
>>> 2012-06-20 10:35:52,612 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
>>> 127.0.0.1:46240 => /127.0.0.1:41414] BOUND: /127.0.0.1:41414
>>> 2012-06-20 10:35:52,612 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
>>> 127.0.0.1:46240 => /127.0.0.1:41414] CONNECTED: /127.0.0.1:46240
>>> 2012-06-20 10:35:52,792 WARN ipc.NettyServer: Unexpected exception from
>>> downstream.
>>> java.lang.OutOfMemoryError: Java heap space
>>> at java.util.ArrayList.<init>(ArrayList.java:112)
>>> at
>>> org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:154)
>>> at
>>> org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:131)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> Is there something else i need to perform?
>>>
>>> Regards,
>>> Som Shekhar Sharma
>>>
>>>
>>>
>>
>
Re: Unable to transport the event to Avro Source
Posted by shekhar sharma <sh...@gmail.com>.
Hari, can you suggest me what needs to be done to convert the events into
Avro format. I am also trying to find out, but it would be great if you can
give me some pointers to start.
Regards,
Som
On Wed, Jun 20, 2012 at 11:30 PM, shekhar sharma <sh...@gmail.com>wrote:
> Thanks hari for prompt reply. I dont have a sink that writes to avro
> source.
> Actually i am writing a external program which will send the events to the
> Avro Source. you mean to say, my external program should convert the events
> into avro format and then i can send those event through a socket.
>
> Regards,
> SOm
>
>
> On Wed, Jun 20, 2012 at 11:22 PM, Hari Shreedharan <
> hshreedharan@cloudera.com> wrote:
>
>> Hi Som,
>>
>> AvroSource is designed to received only Avro formatted events, so if you
>> write arbitrary data to it, the AvroSource will behave in an undefined way.
>> If you have a sink that writes to the AvroSource, it should be using the
>> Flume SDK to send the event in the Flume Avro format. If you want to write
>> arbitrary data, I suggest taking a look at the netcat source.
>>
>> Thanks
>> Hari
>>
>> --
>> Hari Shreedharan
>>
>> On Wednesday, June 20, 2012 at 10:48 AM, shekhar sharma wrote:
>>
>> Hello,
>> I am encountering an exception when i am trying to transport the events
>> to flume avro source.
>> Basically i have created an agent consisting of avro source and my custom
>> sink.
>> Previously i was using exec source, but due to its limitations i am not
>> using it anymore.
>>
>> Since Avro Source is listening on a port, so i have created a simple
>> application, which opens a socket to that port and send the events through
>> this socket.
>> Socket connection is successful, but when i send the events i got the
>> following error:
>>
>> 2012-06-20 10:34:15,767 INFO source.AvroSource: Avro source
>> starting:AvroSource: { bindAddress:localhost port:41414 }
>> 2012-06-20 10:35:52,588 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
>> 127.0.0.1:46240 => /127.0.0.1:41414] OPEN
>> 2012-06-20 10:35:52,612 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
>> 127.0.0.1:46240 => /127.0.0.1:41414] BOUND: /127.0.0.1:41414
>> 2012-06-20 10:35:52,612 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
>> 127.0.0.1:46240 => /127.0.0.1:41414] CONNECTED: /127.0.0.1:46240
>> 2012-06-20 10:35:52,792 WARN ipc.NettyServer: Unexpected exception from
>> downstream.
>> java.lang.OutOfMemoryError: Java heap space
>> at java.util.ArrayList.<init>(ArrayList.java:112)
>> at
>> org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:154)
>> at
>> org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:131)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
>> at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
>> at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
>> at
>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
>> at
>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
>> at
>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Is there something else i need to perform?
>>
>> Regards,
>> Som Shekhar Sharma
>>
>>
>>
>
Re: Unable to transport the event to Avro Source
Posted by shekhar sharma <sh...@gmail.com>.
Thanks hari for prompt reply. I dont have a sink that writes to avro source.
Actually i am writing a external program which will send the events to the
Avro Source. you mean to say, my external program should convert the events
into avro format and then i can send those event through a socket.
Regards,
SOm
On Wed, Jun 20, 2012 at 11:22 PM, Hari Shreedharan <
hshreedharan@cloudera.com> wrote:
> Hi Som,
>
> AvroSource is designed to received only Avro formatted events, so if you
> write arbitrary data to it, the AvroSource will behave in an undefined way.
> If you have a sink that writes to the AvroSource, it should be using the
> Flume SDK to send the event in the Flume Avro format. If you want to write
> arbitrary data, I suggest taking a look at the netcat source.
>
> Thanks
> Hari
>
> --
> Hari Shreedharan
>
> On Wednesday, June 20, 2012 at 10:48 AM, shekhar sharma wrote:
>
> Hello,
> I am encountering an exception when i am trying to transport the events to
> flume avro source.
> Basically i have created an agent consisting of avro source and my custom
> sink.
> Previously i was using exec source, but due to its limitations i am not
> using it anymore.
>
> Since Avro Source is listening on a port, so i have created a simple
> application, which opens a socket to that port and send the events through
> this socket.
> Socket connection is successful, but when i send the events i got the
> following error:
>
> 2012-06-20 10:34:15,767 INFO source.AvroSource: Avro source
> starting:AvroSource: { bindAddress:localhost port:41414 }
> 2012-06-20 10:35:52,588 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
> 127.0.0.1:46240 => /127.0.0.1:41414] OPEN
> 2012-06-20 10:35:52,612 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
> 127.0.0.1:46240 => /127.0.0.1:41414] BOUND: /127.0.0.1:41414
> 2012-06-20 10:35:52,612 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
> 127.0.0.1:46240 => /127.0.0.1:41414] CONNECTED: /127.0.0.1:46240
> 2012-06-20 10:35:52,792 WARN ipc.NettyServer: Unexpected exception from
> downstream.
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.<init>(ArrayList.java:112)
> at
> org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:154)
> at
> org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:131)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Is there something else i need to perform?
>
> Regards,
> Som Shekhar Sharma
>
>
>
Re: Unable to transport the event to Avro Source
Posted by Will McQueen <wi...@cloudera.com>.
To add to what Hari says, you can try either creating your own avro client
using the Flume SDK (eg, you can grab the default RpcClient from the
factory class) or you could try using the avro-client (eg, bin/flume-ng
avro-client...). If you'd like to use the SDK, I suggest taking a look at:
1) The JavaDocs
2) The tests under flume-ng-sdk module (for executable documentation)
3) The source code under flume-ng-sdk module (to drill down deeper if
desired)
Cheers,
Will
On Wed, Jun 20, 2012 at 10:52 AM, Hari Shreedharan <
hshreedharan@cloudera.com> wrote:
> Hi Som,
>
> AvroSource is designed to received only Avro formatted events, so if you
> write arbitrary data to it, the AvroSource will behave in an undefined way.
> If you have a sink that writes to the AvroSource, it should be using the
> Flume SDK to send the event in the Flume Avro format. If you want to write
> arbitrary data, I suggest taking a look at the netcat source.
>
> Thanks
> Hari
>
> --
> Hari Shreedharan
>
> On Wednesday, June 20, 2012 at 10:48 AM, shekhar sharma wrote:
>
> Hello,
> I am encountering an exception when i am trying to transport the events to
> flume avro source.
> Basically i have created an agent consisting of avro source and my custom
> sink.
> Previously i was using exec source, but due to its limitations i am not
> using it anymore.
>
> Since Avro Source is listening on a port, so i have created a simple
> application, which opens a socket to that port and send the events through
> this socket.
> Socket connection is successful, but when i send the events i got the
> following error:
>
> 2012-06-20 10:34:15,767 INFO source.AvroSource: Avro source
> starting:AvroSource: { bindAddress:localhost port:41414 }
> 2012-06-20 10:35:52,588 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
> 127.0.0.1:46240 => /127.0.0.1:41414] OPEN
> 2012-06-20 10:35:52,612 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
> 127.0.0.1:46240 => /127.0.0.1:41414] BOUND: /127.0.0.1:41414
> 2012-06-20 10:35:52,612 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
> 127.0.0.1:46240 => /127.0.0.1:41414] CONNECTED: /127.0.0.1:46240
> 2012-06-20 10:35:52,792 WARN ipc.NettyServer: Unexpected exception from
> downstream.
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.<init>(ArrayList.java:112)
> at
> org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:154)
> at
> org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:131)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Is there something else i need to perform?
>
> Regards,
> Som Shekhar Sharma
>
>
>
Re: Unable to transport the event to Avro Source
Posted by Hari Shreedharan <hs...@cloudera.com>.
Hi Som,
AvroSource is designed to received only Avro formatted events, so if you write arbitrary data to it, the AvroSource will behave in an undefined way. If you have a sink that writes to the AvroSource, it should be using the Flume SDK to send the event in the Flume Avro format. If you want to write arbitrary data, I suggest taking a look at the netcat source.
Thanks
Hari
--
Hari Shreedharan
On Wednesday, June 20, 2012 at 10:48 AM, shekhar sharma wrote:
> Hello,
> I am encountering an exception when i am trying to transport the events to flume avro source.
> Basically i have created an agent consisting of avro source and my custom sink.
> Previously i was using exec source, but due to its limitations i am not using it anymore.
>
> Since Avro Source is listening on a port, so i have created a simple application, which opens a socket to that port and send the events through this socket.
> Socket connection is successful, but when i send the events i got the following error:
>
> 2012-06-20 10:34:15,767 INFO source.AvroSource: Avro source starting:AvroSource: { bindAddress:localhost port:41414 }
> 2012-06-20 10:35:52,588 INFO ipc.NettyServer: [id: 0x00f0b4a3, /127.0.0.1:46240 (http://127.0.0.1:46240) => /127.0.0.1:41414 (http://127.0.0.1:41414)] OPEN
> 2012-06-20 10:35:52,612 INFO ipc.NettyServer: [id: 0x00f0b4a3, /127.0.0.1:46240 (http://127.0.0.1:46240) => /127.0.0.1:41414 (http://127.0.0.1:41414)] BOUND: /127.0.0.1:41414 (http://127.0.0.1:41414)
> 2012-06-20 10:35:52,612 INFO ipc.NettyServer: [id: 0x00f0b4a3, /127.0.0.1:46240 (http://127.0.0.1:46240) => /127.0.0.1:41414 (http://127.0.0.1:41414)] CONNECTED: /127.0.0.1:46240 (http://127.0.0.1:46240)
> 2012-06-20 10:35:52,792 WARN ipc.NettyServer: Unexpected exception from downstream.
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.<init>(ArrayList.java:112)
> at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:154)
> at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:131)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
> at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Is there something else i need to perform?
>
> Regards,
> Som Shekhar Sharma
>