[jira] [Created] (FLUME-3211) Byte capacity seems limiting the flume kafka source operation.

["kaushik srinivas (JIRA)" <ji...@apache.org>]

kaushik srinivas created FLUME-3211:
---------------------------------------

             Summary: Byte capacity seems limiting the flume kafka source operation.
                 Key: FLUME-3211
                 URL: https://issues.apache.org/jira/browse/FLUME-3211
             Project: Flume
          Issue Type: Question
            Reporter: kaushik srinivas
            Priority: Critical


Flume kafka source has been used with spillable memory channel to dump messages into kite dataset.

Every message from a kafka topic is around 180 bytes.
And messages/sec is approx 80k/sec.

So, flume spill-able memory channel has been configured with,
memory capacity : 5000000
transaction capacity : 50000
batch size : 50000
byteCapacityBufferPercentage : 30

Upon addition of parameter byteCapacity with value 900000000 (memoryCapacity * size of each message/event i.e 5000000*180),
We see the kafka bytes out/sec due to flume kafka source is very less compared to kafka bytes in/sec.

If byteCapacity is removed from flume config, the bytesOut/sec seems to increase and equal to bytesIn/sec.

What is the purpose of having byteCapacity and what should be the right way to calculate the same ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org