You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by James Stewart <Ja...@Optiver.com.au> on 2013/03/12 00:53:39 UTC
FLume OG Choke Limit Not Working
Hello all,
I'm using flume OG (unable to upgrade to NG at this stage) and I am having trouble with the choke decorator.
I am aggregating the data flows from several logical nodes at a single 'aggregator' logical node. The data flows should be batched, zipped, choked and then sent on to another 'collector' logical node. I am using the following config to achieve this:
exec setChokeLimit aggregator.mydomain.com mychoke 150
exec config aggregator.mydomain.com 'collectorSource(35853)' 'batch(100, 1000) gzip choke("mychoke") agentBESink("collector.mydomain.com", 35853)'
The choke decorator should limit transfer to 150KB/sec, which equates to 1.2Mbit/sec. However I am regularly recording Flume traffic spikes of 5Mbit/sec and more.
Can anybody suggest what I might be doing wrong? Is it ok to chain the batch, gzip and choke decorators like this, or should they each be in a separate logical node?
Thanks,
James
________________________________
Information contained in this communication (including any attachments) is confidential and may be privileged or subject to copyright. If you have received this communication in error you are not authorised to use the information in any way and Optiver requests that you notify the sender by return email, destroy all copies and delete the information from your system. Optiver does not represent, warrant or guarantee that this communication is free from computer viruses or other defects or that the integrity of this communication has been maintained. Any views expressed in this communication are those of the individual sender. Optiver does not accept liability for any loss or damage caused directly or indirectly by this communication or its use.
Please consider the environment before printing this email.
Re: FLume OG Choke Limit Not Working
Posted by Jeong-shik Jang <js...@gruter.com>.
Hi James,
According to source codes, depending on event body size, choke limit may
not work accurately.
One scenario I can imagine for this case is:
* Given choke limit 10 which means 1 KB bucket size and event body
size 100KB,
* when append() is called in choke deco, it may wait for 2 times (200
milliseconds) as 100KB is larger than bucket size (1 KB)
* but at third time it just ships the event to Sink no matter how
large the event body is,
* this can happen about 3 times within a seconds (every 100 * 3
milliseconds).
* This may result in a spike of more than 300KB (100KB * 3)/sec;
2.4Mbit/sec.
So I am curious about the size of your event body.
-JS
On 3/13/13 7:35 PM, James Stewart wrote:
>
> Hello again,
>
> Further to my previous problem with the Choke decorator under Flume
> OG, I've tried separating out my 'aggregator' into 3 logical nodes as
> follows...
>
> OLD:
>
> exec setChokeLimit aggregator.mydomain.com mychoke 150
>
> exec config aggregator.mydomain.com 'collectorSource(35853)'
> 'batch(100, 1000) gzip choke("mychoke")
> agentBESink("collector.mydomain.com", 35853)'
>
> NEW:
>
> exec setChokeLimit aggregator.mydomain.com mychoke 150
>
> exec config batcher 'collectorSource(35853)' 'batch(100, 1000)
> agentBESink("aggregator.mydomain.com", 35854)'
>
> exec config zipper 'collectorSource(35854)' 'gzip
> agentBESink("aggregator.mydomain.com ", 35855)'
>
> exec config choker 'collectorSource(35855)' 'choke("mychoke")
> agentBESink("collector.mydomain.com ", 35853)'
>
> Looking at the stats, it seems that the 'choker' decorator is having
> no effect on the data passing through it. Even when I set 'mychoke' to
> 10, (which equates to 80Kbit/sec) I still see regular spikes above
> 1.5Mbit/sec.
>
> The Flume User Guide says:
>
> /"The choke decorator works as follows: when append() is called on the
> sink to which the choke is attached, the append() call works normally
> if the amount of data transferred (during a small duration of time) is
> within the limit assigned to the choke-id corresponding to the choke.
> If the limit has been exceeded, then append() is blocked for a small
> duration of time."/
>
> Our traffic is quite bursty -- could this be a problem for the Choke
> decorator?
>
> Any help is much appreciated as I've hit a bit of a wall.
>
> Cheers,
>
> James
>
> *From:*James Stewart [mailto:James.Stewart@Optiver.com.au]
> *Sent:* Tuesday, 12 March 2013 10:54 AM
> *To:* user@flume.apache.org
> *Subject:* FLume OG Choke Limit Not Working
>
> Hello all,
>
> I'm using flume OG (unable to upgrade to NG at this stage) and I am
> having trouble with the choke decorator.
>
> I am aggregating the data flows from several logical nodes at a single
> 'aggregator' logical node. The data flows should be batched, zipped,
> choked and then sent on to another 'collector' logical node. I am
> using the following config to achieve this:
>
> exec setChokeLimit aggregator.mydomain.com mychoke 150
>
> exec config aggregator.mydomain.com 'collectorSource(35853)'
> 'batch(100, 1000) gzip choke("mychoke")
> agentBESink("collector.mydomain.com", 35853)'
>
> The choke decorator should limit transfer to 150KB/sec, which equates
> to 1.2Mbit/sec. However I am regularly recording Flume traffic spikes
> of 5Mbit/sec and more.
>
> Can anybody suggest what I might be doing wrong? Is it ok to chain the
> batch, gzip and choke decorators like this, or should they each be in
> a separate logical node?
>
> Thanks,
>
> James
>
> ------------------------------------------------------------------------
>
> Information contained in this communication (including any
> attachments) is confidential and may be privileged or subject to
> copyright. If you have received this communication in error you are
> not authorised to use the information in any way and Optiver requests
> that you notify the sender by return email, destroy all copies and
> delete the information from your system. Optiver does not represent,
> warrant or guarantee that this communication is free from computer
> viruses or other defects or that the integrity of this communication
> has been maintained. Any views expressed in this communication are
> those of the individual sender. Optiver does not accept liability for
> any loss or damage caused directly or indirectly by this communication
> or its use.
>
> Please consider the environment before printing this email.
>
--
Jeong-shik Jang / jsjang@gruter.com
Gruter, Inc., R&D Team Leader
www.gruter.com
Enjoy Connecting
RE: FLume OG Choke Limit Not Working
Posted by James Stewart <Ja...@Optiver.com.au>.
Hello again,
Further to my previous problem with the Choke decorator under Flume OG, I've tried separating out my 'aggregator' into 3 logical nodes as follows...
OLD:
exec setChokeLimit aggregator.mydomain.com mychoke 150
exec config aggregator.mydomain.com 'collectorSource(35853)' 'batch(100, 1000) gzip choke("mychoke") agentBESink("collector.mydomain.com", 35853)'
NEW:
exec setChokeLimit aggregator.mydomain.com mychoke 150
exec config batcher 'collectorSource(35853)' 'batch(100, 1000) agentBESink("aggregator.mydomain.com", 35854)'
exec config zipper 'collectorSource(35854)' 'gzip agentBESink("aggregator.mydomain.com ", 35855)'
exec config choker 'collectorSource(35855)' 'choke("mychoke") agentBESink("collector.mydomain.com ", 35853)'
Looking at the stats, it seems that the 'choker' decorator is having no effect on the data passing through it. Even when I set 'mychoke' to 10, (which equates to 80Kbit/sec) I still see regular spikes above 1.5Mbit/sec.
The Flume User Guide says:
"The choke decorator works as follows: when append() is called on the sink to which the choke is attached, the append() call works normally if the amount of data transferred (during a small duration of time) is within the limit assigned to the choke-id corresponding to the choke. If the limit has been exceeded, then append() is blocked for a small duration of time."
Our traffic is quite bursty - could this be a problem for the Choke decorator?
Any help is much appreciated as I've hit a bit of a wall.
Cheers,
James
From: James Stewart [mailto:James.Stewart@Optiver.com.au]
Sent: Tuesday, 12 March 2013 10:54 AM
To: user@flume.apache.org
Subject: FLume OG Choke Limit Not Working
Hello all,
I'm using flume OG (unable to upgrade to NG at this stage) and I am having trouble with the choke decorator.
I am aggregating the data flows from several logical nodes at a single 'aggregator' logical node. The data flows should be batched, zipped, choked and then sent on to another 'collector' logical node. I am using the following config to achieve this:
exec setChokeLimit aggregator.mydomain.com mychoke 150
exec config aggregator.mydomain.com 'collectorSource(35853)' 'batch(100, 1000) gzip choke("mychoke") agentBESink("collector.mydomain.com", 35853)'
The choke decorator should limit transfer to 150KB/sec, which equates to 1.2Mbit/sec. However I am regularly recording Flume traffic spikes of 5Mbit/sec and more.
Can anybody suggest what I might be doing wrong? Is it ok to chain the batch, gzip and choke decorators like this, or should they each be in a separate logical node?
Thanks,
James
________________________________
Information contained in this communication (including any attachments) is confidential and may be privileged or subject to copyright. If you have received this communication in error you are not authorised to use the information in any way and Optiver requests that you notify the sender by return email, destroy all copies and delete the information from your system. Optiver does not represent, warrant or guarantee that this communication is free from computer viruses or other defects or that the integrity of this communication has been maintained. Any views expressed in this communication are those of the individual sender. Optiver does not accept liability for any loss or damage caused directly or indirectly by this communication or its use.
Please consider the environment before printing this email.