You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Rahul Ravindran <ra...@yahoo.com> on 2012/11/09 00:05:14 UTC
Netcat source stops processing data
Hello,
I wanted to perform a load test to get an idea of how we would look to scale flume for our deployment. I have pasted the config file at the source below. I have a netcat source which is listening on a port and have 2 channels, 2 avro sinks consuming the events from the netcat source.
My load generator is a simple C program which is continually sending 20 characters in each message using a socket, and send(). I notice that , initially, a lot of traffic makes it through and then the flume agent appears to stop consuming data(after about 80k messages). This results in the tcp receive and send buffer being full. I understand that the rate at which I am generating traffic may overwhelm flume, but I would expect it to gradually consume data. It does not consume any more messages. I looked through the flume logs and did not see anything there (no stack trace). I ran tcpdump and see that the receive window initially is non-zero but begins to decrease and then goes down to zero, and very slowly opens up to a size of 1 (once in 10 seconds)
Could you help on what may be going on or if there is something wrong with my config?
agent1.channels.ch1.type = MEMORY
agent1.channels.ch1.capacity = 50000
agent1.channels.ch1.transactionCapacity = 5000
agent1.sources.netcat.channels = ch1
agent1.sources.netcat.type= netcat
agent1.sources.netcat.bind = 127.0.0.1
agent1.sources.netcat.port = 44444
agent1.sinks.avroSink1.type = avro
agent1.sinks.avroSink1.channel = ch1
agent1.sinks.avroSink1.hostname = <remote hostname>
agent1.sinks.avroSink1.port = 4545
agent1.sinks.avroSink1.connect-timeout = 300000
agent1.sinks.avroSink2.type = avro
agent1.sinks.avroSink2.channel = ch1
agent1.sinks.avroSink2.hostname = <remote hostname>
agent1.sinks.avroSink2.port = 4546
agent1.sinks.avroSink2.connect-timeout = 300000
agent1.channels = ch1
agent1.sources = netcat
agent1.sinks = avroSink1 avroSink2 avroSink2
Re: Netcat source stops processing data
Posted by Mike Percy <mp...@apache.org>.
Rahul,
A patch and a unit test to add this as an option would be greatly
appreciated!
There is already a JIRA open for this:
https://issues.apache.org/jira/browse/FLUME-1713
Regards,
Mike
On Tue, Nov 20, 2012 at 3:20 PM, Rahul Ravindran <ra...@yahoo.com> wrote:
> Pinging on this slightly old thread.
>
> I want to avoid the overhead of HTTP. Is it possible to get the NetCat
> source be fixed to not send the OK? I did a local build of flume without
> the "OK" being written, and it works fine.
> If there are tests which depend on the OK being sent back, could an
> additional parameter which skips it be added?
>
>
>
> ------------------------------
> *From:* Rahul Ravindran <ra...@yahoo.com>
> *To:* "user@flume.apache.org" <us...@flume.apache.org>
> *Cc:* "user@flume.apache.org" <us...@flume.apache.org>
> *Sent:* Thursday, November 8, 2012 6:15 PM
> *Subject:* Re: Netcat source stops processing data
>
> Thanks! Will try removing the ok.
>
> Sent from my phone.Excuse the terseness.
>
> On Nov 8, 2012, at 3:15 PM, Hari Shreedharan <hs...@cloudera.com>
> wrote:
>
> Rahul,
>
> Are you reading the responses sent by the net cat source? If you don't
> read the OK sent by net cat source on your application side, your
> application's buffer gets full causing the net cat source to queue up stuff
> and eventually die. It is something we need to fix, but I don't know if
> anyone is using net cat in production - you should probably test using Avro
> source or the new HTTP source(for this you would need to build trunk/1.3
> branch or wait for 1.3 release).
>
>
> Thanks
> Hari
>
> --
> Hari Shreedharan
>
> On Thursday, November 8, 2012 at 3:05 PM, Rahul Ravindran wrote:
>
> Hello,
> I wanted to perform a load test to get an idea of how we would look to
> scale flume for our deployment. I have pasted the config file at the source
> below. I have a netcat source which is listening on a port and have 2
> channels, 2 avro sinks consuming the events from the netcat source.
>
> My load generator is a simple C program which is continually sending 20
> characters in each message using a socket, and send(). I notice that ,
> initially, a lot of traffic makes it through and then the flume agent
> appears to stop consuming data(after about 80k messages). This results in
> the tcp receive and send buffer being full. I understand that the rate at
> which I am generating traffic may overwhelm flume, but I would expect it to
> gradually consume data. It does not consume any more messages. I looked
> through the flume logs and did not see anything there (no stack trace). I
> ran tcpdump and see that the receive window initially is non-zero but
> begins to decrease and then goes down to zero, and very slowly opens up to
> a size of 1 (once in 10 seconds)
>
> Could you help on what may be going on or if there is something wrong with
> my config?
>
> agent1.channels.ch1.type = MEMORY
> agent1.channels.ch1.capacity = 50000
> agent1.channels.ch1.transactionCapacity = 5000
>
> agent1.sources.netcat.channels = ch1
> agent1.sources.netcat.type= netcat
> agent1.sources.netcat.bind = 127.0.0.1
> agent1.sources.netcat.port = 44444
>
> agent1.sinks.avroSink1.type = avro
> agent1.sinks.avroSink1.channel = ch1
> agent1.sinks.avroSink1.hostname = <remote hostname>
> agent1.sinks.avroSink1.port = 4545
> agent1.sinks.avroSink1.connect-timeout = 300000
>
>
> agent1.sinks.avroSink2.type = avro
> agent1.sinks.avroSink2.channel = ch1
> agent1.sinks.avroSink2.hostname = <remote hostname>
> agent1.sinks.avroSink2.port = 4546
> agent1.sinks.avroSink2.connect-timeout = 300000
>
> agent1.channels = ch1
> agent1.sources = netcat
> agent1.sinks = avroSink1 avroSink2 avroSink2
>
>
>
>
>
Re: Netcat source stops processing data
Posted by Rahul Ravindran <ra...@yahoo.com>.
Pinging on this slightly old thread.
I want to avoid the overhead of HTTP. Is it possible to get the NetCat source be fixed to not send the OK? I did a local build of flume without the "OK" being written, and it works fine.
If there are tests which depend on the OK being sent back, could an additional parameter which skips it be added?
________________________________
From: Rahul Ravindran <ra...@yahoo.com>
To: "user@flume.apache.org" <us...@flume.apache.org>
Cc: "user@flume.apache.org" <us...@flume.apache.org>
Sent: Thursday, November 8, 2012 6:15 PM
Subject: Re: Netcat source stops processing data
Thanks! Will try removing the ok.
Sent from my phone.Excuse the terseness.
On Nov 8, 2012, at 3:15 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
Rahul,
>
>
>Are you reading the responses sent by the net cat source? If you don't read the OK sent by net cat source on your application side, your application's buffer gets full causing the net cat source to queue up stuff and eventually die. It is something we need to fix, but I don't know if anyone is using net cat in production - you should probably test using Avro source or the new HTTP source(for this you would need to build trunk/1.3 branch or wait for 1.3 release).
>
>
>
>
>Thanks
>Hari
>
>
>--
>Hari Shreedharan
>
>
>On Thursday, November 8, 2012 at 3:05 PM, Rahul Ravindran wrote:
>Hello,
>> I wanted to perform a load test to get an idea of how we would look to scale flume for our deployment. I have pasted the config file at the source below. I have a netcat source which is listening on a port and have 2 channels, 2 avro sinks consuming the events from the netcat source.
>>
>>
>>My load generator is a simple C program which is continually sending 20 characters in each message using a socket, and send(). I notice that , initially, a lot of traffic makes it through and then the flume agent appears to stop consuming data(after about 80k messages). This results in the tcp receive and send buffer being full. I understand that the rate at which I am generating traffic may overwhelm flume, but I would expect it to gradually consume data. It does not consume any more messages. I looked through the flume logs and did not see anything there (no stack trace). I ran tcpdump and see that the receive window initially is non-zero but begins to decrease and then goes down to zero, and very slowly opens up to a size of 1 (once in 10 seconds)
>>
>>
>>Could you help on what may be going on or if there is something wrong with my config?
>>
>>
>>agent1.channels.ch1.type = MEMORY
>>agent1.channels.ch1.capacity = 50000
>>agent1.channels.ch1.transactionCapacity = 5000
>>
>>
>>agent1.sources.netcat.channels = ch1
>>agent1.sources.netcat.type= netcat
>>agent1.sources.netcat.bind = 127.0.0.1
>>agent1.sources.netcat.port = 44444
>>
>>
>>agent1.sinks.avroSink1.type = avro
>>agent1.sinks.avroSink1.channel = ch1
>>agent1.sinks.avroSink1.hostname = <remote hostname>
>>agent1.sinks.avroSink1.port = 4545
>>agent1.sinks.avroSink1.connect-timeout = 300000
>>
>>
>>
>>
>>agent1.sinks.avroSink2.type = avro
>>agent1.sinks.avroSink2.channel = ch1
>>agent1.sinks.avroSink2.hostname = <remote hostname>
>>agent1.sinks.avroSink2.port = 4546
>>agent1.sinks.avroSink2.connect-timeout = 300000
>>
>>
>>agent1.channels = ch1
>>agent1.sources = netcat
>>agent1.sinks = avroSink1 avroSink2 avroSink2
>
>
Re: Netcat source stops processing data
Posted by Rahul Ravindran <ra...@yahoo.com>.
Thanks! Will try removing the ok.
Sent from my phone.Excuse the terseness.
On Nov 8, 2012, at 3:15 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
> Rahul,
>
> Are you reading the responses sent by the net cat source? If you don't read the OK sent by net cat source on your application side, your application's buffer gets full causing the net cat source to queue up stuff and eventually die. It is something we need to fix, but I don't know if anyone is using net cat in production - you should probably test using Avro source or the new HTTP source(for this you would need to build trunk/1.3 branch or wait for 1.3 release).
>
>
> Thanks
> Hari
>
> --
> Hari Shreedharan
>
> On Thursday, November 8, 2012 at 3:05 PM, Rahul Ravindran wrote:
>
>> Hello,
>> I wanted to perform a load test to get an idea of how we would look to scale flume for our deployment. I have pasted the config file at the source below. I have a netcat source which is listening on a port and have 2 channels, 2 avro sinks consuming the events from the netcat source.
>>
>> My load generator is a simple C program which is continually sending 20 characters in each message using a socket, and send(). I notice that , initially, a lot of traffic makes it through and then the flume agent appears to stop consuming data(after about 80k messages). This results in the tcp receive and send buffer being full. I understand that the rate at which I am generating traffic may overwhelm flume, but I would expect it to gradually consume data. It does not consume any more messages. I looked through the flume logs and did not see anything there (no stack trace). I ran tcpdump and see that the receive window initially is non-zero but begins to decrease and then goes down to zero, and very slowly opens up to a size of 1 (once in 10 seconds)
>>
>> Could you help on what may be going on or if there is something wrong with my config?
>>
>> agent1.channels.ch1.type = MEMORY
>> agent1.channels.ch1.capacity = 50000
>> agent1.channels.ch1.transactionCapacity = 5000
>>
>> agent1.sources.netcat.channels = ch1
>> agent1.sources.netcat.type= netcat
>> agent1.sources.netcat.bind = 127.0.0.1
>> agent1.sources.netcat.port = 44444
>>
>> agent1.sinks.avroSink1.type = avro
>> agent1.sinks.avroSink1.channel = ch1
>> agent1.sinks.avroSink1.hostname = <remote hostname>
>> agent1.sinks.avroSink1.port = 4545
>> agent1.sinks.avroSink1.connect-timeout = 300000
>>
>>
>> agent1.sinks.avroSink2.type = avro
>> agent1.sinks.avroSink2.channel = ch1
>> agent1.sinks.avroSink2.hostname = <remote hostname>
>> agent1.sinks.avroSink2.port = 4546
>> agent1.sinks.avroSink2.connect-timeout = 300000
>>
>> agent1.channels = ch1
>> agent1.sources = netcat
>> agent1.sinks = avroSink1 avroSink2 avroSink2
>
Re: Netcat source stops processing data
Posted by Hari Shreedharan <hs...@cloudera.com>.
Rahul,
Are you reading the responses sent by the net cat source? If you don't read the OK sent by net cat source on your application side, your application's buffer gets full causing the net cat source to queue up stuff and eventually die. It is something we need to fix, but I don't know if anyone is using net cat in production - you should probably test using Avro source or the new HTTP source(for this you would need to build trunk/1.3 branch or wait for 1.3 release).
Thanks
Hari
--
Hari Shreedharan
On Thursday, November 8, 2012 at 3:05 PM, Rahul Ravindran wrote:
> Hello,
> I wanted to perform a load test to get an idea of how we would look to scale flume for our deployment. I have pasted the config file at the source below. I have a netcat source which is listening on a port and have 2 channels, 2 avro sinks consuming the events from the netcat source.
>
> My load generator is a simple C program which is continually sending 20 characters in each message using a socket, and send(). I notice that , initially, a lot of traffic makes it through and then the flume agent appears to stop consuming data(after about 80k messages). This results in the tcp receive and send buffer being full. I understand that the rate at which I am generating traffic may overwhelm flume, but I would expect it to gradually consume data. It does not consume any more messages. I looked through the flume logs and did not see anything there (no stack trace). I ran tcpdump and see that the receive window initially is non-zero but begins to decrease and then goes down to zero, and very slowly opens up to a size of 1 (once in 10 seconds)
>
> Could you help on what may be going on or if there is something wrong with my config?
>
> agent1.channels.ch1.type = MEMORY
> agent1.channels.ch1.capacity = 50000
> agent1.channels.ch1.transactionCapacity = 5000
>
> agent1.sources.netcat.channels = ch1
> agent1.sources.netcat.type= netcat
> agent1.sources.netcat.bind = 127.0.0.1
> agent1.sources.netcat.port = 44444
>
> agent1.sinks.avroSink1.type = avro
> agent1.sinks.avroSink1.channel = ch1
> agent1.sinks.avroSink1.hostname = <remote hostname>
> agent1.sinks.avroSink1.port = 4545
> agent1.sinks.avroSink1.connect-timeout = 300000
>
>
> agent1.sinks.avroSink2.type = avro
> agent1.sinks.avroSink2.channel = ch1
> agent1.sinks.avroSink2.hostname = <remote hostname>
> agent1.sinks.avroSink2.port = 4546
> agent1.sinks.avroSink2.connect-timeout = 300000
>
> agent1.channels = ch1
> agent1.sources = netcat
> agent1.sinks = avroSink1 avroSink2 avroSink2
>
>
>
>