You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Bernard Quizon <be...@stellarloyalty.com> on 2019/01/07 07:29:27 UTC
Kafka-Storm Performance Tests
Hi,
Before anything else, I'm using Apache Storm 1.2.1 and Apache Kafka 1.1.1.
On my initial tests, I have:
- *6 supervisor nodes *
- 1 worker per node
- 7 KafkaSpout executors per worker
- 1 bolt (that does nothing) per worker
- 0 Ackers
- 2 Kafka brokers with *42 partitions *on the topic
With that configuration, no matter how I change some other configs (Please
see list below), my topology is capping out at around 48k TPS.
Please note that CPU usage, for the Supervisor nodes, is only around 20%
and Network usage is only around 20Mbps for both the Kafka Brokers and
Supervisor nodes, well below the network capacities.
Now, I have increased the supervisor nodes from 6 to 12 and used a new
topic with 82 partitions, thinking that scaling out could help the
performance.
So is this the new configuration:
- *12 supervisor nodes *
- 1 worker per node
- 7 KafkaSpout executors per worker
- 1 bolt (that does nothing) per worker
- 0 Ackers
- 2 Kafka brokers with *84 partitions* on the topic
And I'm still getting around 48k TPS.
Some other configs I played around with:
- max.poll.records
- max.spout.pending
- processing.guarantee
- offset.commit.period.ms
- max.uncommitted.offsets
- poll.timeout.ms
- fetch.min.bytes
- fetch.max.bytes
- max.partition.fetch.bytes
- receive.buffer.bytes
- fetch.max.wait.ms
- topology.executor.receive.buffer.size
- topology.executor.send.buffer.size
- topology.receiver.buffer.size
- topology.transfer.buffer.size
Am I missing something here? Why is the throughput not improving?
Just to add, I have also done some performance isolation tests on both
Kafka and Storm.
On a distributed consumer using Spark and Kafka, I was able to get around
700K TPS (So we know that Kafka isn't the issue here).
And also I could get around 400k TPS on a custom Storm topology with 1
spout that generates random transactions and 1 bolt that does nothing.
I feel like the numbers don't add up and the topology shouldn't be capping
out at around 48K TPS.
Your suggestions would be very much appreciated.
Thanks,
Bernard
Re: Kafka-Storm Performance Tests
Posted by Roshan Naik <ro...@yahoo.com>.
I suspect having only one bolt per worker may be holding things a back a bit. Also check if enabling/disabling load aware messaging makes any diff.
BTW..what is your avg event size ?
Sent from Yahoo Mail for iPhone
On Tuesday, January 8, 2019, 7:06 PM, Bernard Quizon <be...@stellarloyalty.com> wrote:
Hi,
I've tried using the patched `storm-kafka-client` and my throughput was up from 48K TPS to 585K TPS using my 6 supervisor node setup.Thank you Stig and Roshan.
-Bernard
On Mon, Jan 7, 2019 at 5:47 PM Bernard Quizon <be...@stellarloyalty.com> wrote:
Wow, thanks Stig! That's very helpful.
On Mon, Jan 7, 2019 at 4:50 PM Stig Rohde Døssing <st...@gmail.com> wrote:
I'm not sure if it's the only issue, but you might be affected by https://issues.apache.org/jira/browse/STORM-3102.
Den man. 7. jan. 2019 kl. 09.37 skrev Bernard Quizon <be...@stellarloyalty.com>:
Hi Roshan,
Just to narrow it down .. can u get rid of the bolt and see what numbers you get ?
> Thanks, I will try that.
Which Kafka spout are you using (new/old) ?
> The new one, from kafka-clients.
I assume you mean msgs when you say ‘transactions’. > Yes that would be messages
Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per worker ?
> That would be the throughput of the whole topology, not per worker.
Thanks,
Bernard
On Mon, Jan 7, 2019 at 4:24 PM Roshan Naik <ro...@yahoo.com> wrote:
Just to narrow it down .. can u get rid of the bolt and see what numbers you get ?
Which Kafka spout are you using (new/old) ?
I assume you mean msgs when you say ‘transactions’. Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per worker ?
Sent from Yahoo Mail for iPhone
On Sunday, January 6, 2019, 11:29 PM, Bernard Quizon <be...@stellarloyalty.com> wrote:
Hi,
Before anything else, I'm using Apache Storm 1.2.1 and Apache Kafka 1.1.1.
On my initial tests, I have:
- 6 supervisor nodes
- 1 worker per node
- 7 KafkaSpout executors per worker
- 1 bolt (that does nothing) per worker
- 0 Ackers
- 2 Kafka brokers with 42 partitions on the topic
With that configuration, no matter how I change some other configs (Please see list below), my topology is capping out at around 48k TPS.Please note that CPU usage, for the Supervisor nodes, is only around 20% and Network usage is only around 20Mbps for both the Kafka Brokers and Supervisor nodes, well below the network capacities.
Now, I have increased the supervisor nodes from 6 to 12 and used a new topic with 82 partitions, thinking that scaling out could help the performance.So is this the new configuration:
- 12 supervisor nodes
- 1 worker per node
- 7 KafkaSpout executors per worker
- 1 bolt (that does nothing) per worker
- 0 Ackers
- 2 Kafka brokers with 84 partitions on the topic
And I'm still getting around 48k TPS.
Some other configs I played around with:
- max.poll.records
- max.spout.pending
- processing.guarantee
- offset.commit.period.ms
- max.uncommitted.offsets
- poll.timeout.ms
- fetch.min.bytes
- fetch.max.bytes
- max.partition.fetch.bytes
- receive.buffer.bytes
- fetch.max.wait.ms
- topology.executor.receive.buffer.size
- topology.executor.send.buffer.size
- topology.receiver.buffer.size
- topology.transfer.buffer.size
Am I missing something here? Why is the throughput not improving?
Just to add, I have also done some performance isolation tests on both Kafka and Storm.On a distributed consumer using Spark and Kafka, I was able to get around 700K TPS (So we know that Kafka isn't the issue here).And also I could get around 400k TPS on a custom Storm topology with 1 spout that generates random transactions and 1 bolt that does nothing.
I feel like the numbers don't add up and the topology shouldn't be capping out at around 48K TPS.Your suggestions would be very much appreciated.
Thanks,
Bernard
Re: Kafka-Storm Performance Tests
Posted by Bernard Quizon <be...@stellarloyalty.com>.
Hi,
I've tried using the patched `storm-kafka-client` and my throughput was up
from 48K TPS to 585K TPS using my 6 supervisor node setup.
Thank you Stig and Roshan.
-Bernard
On Mon, Jan 7, 2019 at 5:47 PM Bernard Quizon <
bernard.quizon@stellarloyalty.com> wrote:
> Wow, thanks Stig! That's very helpful.
>
> On Mon, Jan 7, 2019 at 4:50 PM Stig Rohde Døssing <st...@gmail.com>
> wrote:
>
>> I'm not sure if it's the only issue, but you might be affected by
>> https://issues.apache.org/jira/browse/STORM-3102.
>>
>> Den man. 7. jan. 2019 kl. 09.37 skrev Bernard Quizon <
>> bernard.quizon@stellarloyalty.com>:
>>
>>> Hi Roshan,
>>>
>>> Just to narrow it down .. can u get rid of the bolt and see what
>>> numbers you get ?
>>> > Thanks, I will try that.
>>>
>>> Which Kafka spout are you using (new/old) ?
>>> > The new one, from kafka-clients.
>>>
>>> I assume you mean msgs when you say ‘transactions’.
>>> > Yes that would be messages
>>>
>>> Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per
>>> worker ?
>>> > That would be the throughput of the whole topology, not per worker.
>>>
>>> Thanks,
>>> Bernard
>>>
>>> On Mon, Jan 7, 2019 at 4:24 PM Roshan Naik <ro...@yahoo.com>
>>> wrote:
>>>
>>>> Just to narrow it down .. can u get rid of the bolt and see what
>>>> numbers you get ?
>>>>
>>>> Which Kafka spout are you using (new/old) ?
>>>>
>>>> I assume you mean msgs when you say ‘transactions’.
>>>> Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per
>>>> worker ?
>>>>
>>>>
>>>>
>>>> Sent from Yahoo Mail for iPhone
>>>> <https://overview.mail.yahoo.com/?.src=iOS>
>>>>
>>>> On Sunday, January 6, 2019, 11:29 PM, Bernard Quizon <
>>>> bernard.quizon@stellarloyalty.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Before anything else, I'm using Apache Storm 1.2.1 and Apache Kafka
>>>> 1.1.1.
>>>> On my initial tests, I have:
>>>>
>>>> - *6 supervisor nodes *
>>>> - 1 worker per node
>>>> - 7 KafkaSpout executors per worker
>>>> - 1 bolt (that does nothing) per worker
>>>> - 0 Ackers
>>>> - 2 Kafka brokers with *42 partitions *on the topic
>>>>
>>>> With that configuration, no matter how I change some other configs
>>>> (Please see list below), my topology is capping out at around 48k TPS.
>>>> Please note that CPU usage, for the Supervisor nodes, is only around
>>>> 20% and Network usage is only around 20Mbps for both the Kafka Brokers and
>>>> Supervisor nodes, well below the network capacities.
>>>>
>>>> Now, I have increased the supervisor nodes from 6 to 12 and used a new
>>>> topic with 82 partitions, thinking that scaling out could help the
>>>> performance.
>>>> So is this the new configuration:
>>>>
>>>> - *12 supervisor nodes *
>>>> - 1 worker per node
>>>> - 7 KafkaSpout executors per worker
>>>> - 1 bolt (that does nothing) per worker
>>>> - 0 Ackers
>>>> - 2 Kafka brokers with *84 partitions* on the topic
>>>>
>>>> And I'm still getting around 48k TPS.
>>>>
>>>> Some other configs I played around with:
>>>>
>>>> - max.poll.records
>>>> - max.spout.pending
>>>> - processing.guarantee
>>>> - offset.commit.period.ms
>>>> - max.uncommitted.offsets
>>>> - poll.timeout.ms
>>>> - fetch.min.bytes
>>>> - fetch.max.bytes
>>>> - max.partition.fetch.bytes
>>>> - receive.buffer.bytes
>>>> - fetch.max.wait.ms
>>>> - topology.executor.receive.buffer.size
>>>> - topology.executor.send.buffer.size
>>>> - topology.receiver.buffer.size
>>>> - topology.transfer.buffer.size
>>>>
>>>>
>>>> Am I missing something here? Why is the throughput not improving?
>>>>
>>>> Just to add, I have also done some performance isolation tests on both
>>>> Kafka and Storm.
>>>> On a distributed consumer using Spark and Kafka, I was able to get
>>>> around 700K TPS (So we know that Kafka isn't the issue here).
>>>> And also I could get around 400k TPS on a custom Storm topology with 1
>>>> spout that generates random transactions and 1 bolt that does nothing.
>>>> I feel like the numbers don't add up and the topology shouldn't be
>>>> capping out at around 48K TPS.
>>>> Your suggestions would be very much appreciated.
>>>>
>>>>
>>>> Thanks,
>>>> Bernard
>>>>
>>>>
Re: Kafka-Storm Performance Tests
Posted by Bernard Quizon <be...@stellarloyalty.com>.
Wow, thanks Stig! That's very helpful.
On Mon, Jan 7, 2019 at 4:50 PM Stig Rohde Døssing <st...@gmail.com>
wrote:
> I'm not sure if it's the only issue, but you might be affected by
> https://issues.apache.org/jira/browse/STORM-3102.
>
> Den man. 7. jan. 2019 kl. 09.37 skrev Bernard Quizon <
> bernard.quizon@stellarloyalty.com>:
>
>> Hi Roshan,
>>
>> Just to narrow it down .. can u get rid of the bolt and see what
>> numbers you get ?
>> > Thanks, I will try that.
>>
>> Which Kafka spout are you using (new/old) ?
>> > The new one, from kafka-clients.
>>
>> I assume you mean msgs when you say ‘transactions’.
>> > Yes that would be messages
>>
>> Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per
>> worker ?
>> > That would be the throughput of the whole topology, not per worker.
>>
>> Thanks,
>> Bernard
>>
>> On Mon, Jan 7, 2019 at 4:24 PM Roshan Naik <ro...@yahoo.com> wrote:
>>
>>> Just to narrow it down .. can u get rid of the bolt and see what
>>> numbers you get ?
>>>
>>> Which Kafka spout are you using (new/old) ?
>>>
>>> I assume you mean msgs when you say ‘transactions’.
>>> Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per
>>> worker ?
>>>
>>>
>>>
>>> Sent from Yahoo Mail for iPhone
>>> <https://overview.mail.yahoo.com/?.src=iOS>
>>>
>>> On Sunday, January 6, 2019, 11:29 PM, Bernard Quizon <
>>> bernard.quizon@stellarloyalty.com> wrote:
>>>
>>> Hi,
>>>
>>> Before anything else, I'm using Apache Storm 1.2.1 and Apache Kafka
>>> 1.1.1.
>>> On my initial tests, I have:
>>>
>>> - *6 supervisor nodes *
>>> - 1 worker per node
>>> - 7 KafkaSpout executors per worker
>>> - 1 bolt (that does nothing) per worker
>>> - 0 Ackers
>>> - 2 Kafka brokers with *42 partitions *on the topic
>>>
>>> With that configuration, no matter how I change some other configs
>>> (Please see list below), my topology is capping out at around 48k TPS.
>>> Please note that CPU usage, for the Supervisor nodes, is only around 20%
>>> and Network usage is only around 20Mbps for both the Kafka Brokers and
>>> Supervisor nodes, well below the network capacities.
>>>
>>> Now, I have increased the supervisor nodes from 6 to 12 and used a new
>>> topic with 82 partitions, thinking that scaling out could help the
>>> performance.
>>> So is this the new configuration:
>>>
>>> - *12 supervisor nodes *
>>> - 1 worker per node
>>> - 7 KafkaSpout executors per worker
>>> - 1 bolt (that does nothing) per worker
>>> - 0 Ackers
>>> - 2 Kafka brokers with *84 partitions* on the topic
>>>
>>> And I'm still getting around 48k TPS.
>>>
>>> Some other configs I played around with:
>>>
>>> - max.poll.records
>>> - max.spout.pending
>>> - processing.guarantee
>>> - offset.commit.period.ms
>>> - max.uncommitted.offsets
>>> - poll.timeout.ms
>>> - fetch.min.bytes
>>> - fetch.max.bytes
>>> - max.partition.fetch.bytes
>>> - receive.buffer.bytes
>>> - fetch.max.wait.ms
>>> - topology.executor.receive.buffer.size
>>> - topology.executor.send.buffer.size
>>> - topology.receiver.buffer.size
>>> - topology.transfer.buffer.size
>>>
>>>
>>> Am I missing something here? Why is the throughput not improving?
>>>
>>> Just to add, I have also done some performance isolation tests on both
>>> Kafka and Storm.
>>> On a distributed consumer using Spark and Kafka, I was able to get
>>> around 700K TPS (So we know that Kafka isn't the issue here).
>>> And also I could get around 400k TPS on a custom Storm topology with 1
>>> spout that generates random transactions and 1 bolt that does nothing.
>>> I feel like the numbers don't add up and the topology shouldn't be
>>> capping out at around 48K TPS.
>>> Your suggestions would be very much appreciated.
>>>
>>>
>>> Thanks,
>>> Bernard
>>>
>>>
Re: Kafka-Storm Performance Tests
Posted by Stig Rohde Døssing <st...@gmail.com>.
I'm not sure if it's the only issue, but you might be affected by
https://issues.apache.org/jira/browse/STORM-3102.
Den man. 7. jan. 2019 kl. 09.37 skrev Bernard Quizon <
bernard.quizon@stellarloyalty.com>:
> Hi Roshan,
>
> Just to narrow it down .. can u get rid of the bolt and see what numbers
> you get ?
> > Thanks, I will try that.
>
> Which Kafka spout are you using (new/old) ?
> > The new one, from kafka-clients.
>
> I assume you mean msgs when you say ‘transactions’.
> > Yes that would be messages
>
> Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per
> worker ?
> > That would be the throughput of the whole topology, not per worker.
>
> Thanks,
> Bernard
>
> On Mon, Jan 7, 2019 at 4:24 PM Roshan Naik <ro...@yahoo.com> wrote:
>
>> Just to narrow it down .. can u get rid of the bolt and see what
>> numbers you get ?
>>
>> Which Kafka spout are you using (new/old) ?
>>
>> I assume you mean msgs when you say ‘transactions’.
>> Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per
>> worker ?
>>
>>
>>
>> Sent from Yahoo Mail for iPhone
>> <https://overview.mail.yahoo.com/?.src=iOS>
>>
>> On Sunday, January 6, 2019, 11:29 PM, Bernard Quizon <
>> bernard.quizon@stellarloyalty.com> wrote:
>>
>> Hi,
>>
>> Before anything else, I'm using Apache Storm 1.2.1 and Apache Kafka 1.1.1.
>> On my initial tests, I have:
>>
>> - *6 supervisor nodes *
>> - 1 worker per node
>> - 7 KafkaSpout executors per worker
>> - 1 bolt (that does nothing) per worker
>> - 0 Ackers
>> - 2 Kafka brokers with *42 partitions *on the topic
>>
>> With that configuration, no matter how I change some other configs
>> (Please see list below), my topology is capping out at around 48k TPS.
>> Please note that CPU usage, for the Supervisor nodes, is only around 20%
>> and Network usage is only around 20Mbps for both the Kafka Brokers and
>> Supervisor nodes, well below the network capacities.
>>
>> Now, I have increased the supervisor nodes from 6 to 12 and used a new
>> topic with 82 partitions, thinking that scaling out could help the
>> performance.
>> So is this the new configuration:
>>
>> - *12 supervisor nodes *
>> - 1 worker per node
>> - 7 KafkaSpout executors per worker
>> - 1 bolt (that does nothing) per worker
>> - 0 Ackers
>> - 2 Kafka brokers with *84 partitions* on the topic
>>
>> And I'm still getting around 48k TPS.
>>
>> Some other configs I played around with:
>>
>> - max.poll.records
>> - max.spout.pending
>> - processing.guarantee
>> - offset.commit.period.ms
>> - max.uncommitted.offsets
>> - poll.timeout.ms
>> - fetch.min.bytes
>> - fetch.max.bytes
>> - max.partition.fetch.bytes
>> - receive.buffer.bytes
>> - fetch.max.wait.ms
>> - topology.executor.receive.buffer.size
>> - topology.executor.send.buffer.size
>> - topology.receiver.buffer.size
>> - topology.transfer.buffer.size
>>
>>
>> Am I missing something here? Why is the throughput not improving?
>>
>> Just to add, I have also done some performance isolation tests on both
>> Kafka and Storm.
>> On a distributed consumer using Spark and Kafka, I was able to get around
>> 700K TPS (So we know that Kafka isn't the issue here).
>> And also I could get around 400k TPS on a custom Storm topology with 1
>> spout that generates random transactions and 1 bolt that does nothing.
>> I feel like the numbers don't add up and the topology shouldn't be
>> capping out at around 48K TPS.
>> Your suggestions would be very much appreciated.
>>
>>
>> Thanks,
>> Bernard
>>
>>
Re: Kafka-Storm Performance Tests
Posted by Bernard Quizon <be...@stellarloyalty.com>.
Hi Roshan,
Just to narrow it down .. can u get rid of the bolt and see what numbers
you get ?
> Thanks, I will try that.
Which Kafka spout are you using (new/old) ?
> The new one, from kafka-clients.
I assume you mean msgs when you say ‘transactions’.
> Yes that would be messages
Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per
worker ?
> That would be the throughput of the whole topology, not per worker.
Thanks,
Bernard
On Mon, Jan 7, 2019 at 4:24 PM Roshan Naik <ro...@yahoo.com> wrote:
> Just to narrow it down .. can u get rid of the bolt and see what numbers
> you get ?
>
> Which Kafka spout are you using (new/old) ?
>
> I assume you mean msgs when you say ‘transactions’.
> Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per
> worker ?
>
>
>
> Sent from Yahoo Mail for iPhone
> <https://overview.mail.yahoo.com/?.src=iOS>
>
> On Sunday, January 6, 2019, 11:29 PM, Bernard Quizon <
> bernard.quizon@stellarloyalty.com> wrote:
>
> Hi,
>
> Before anything else, I'm using Apache Storm 1.2.1 and Apache Kafka 1.1.1.
> On my initial tests, I have:
>
> - *6 supervisor nodes *
> - 1 worker per node
> - 7 KafkaSpout executors per worker
> - 1 bolt (that does nothing) per worker
> - 0 Ackers
> - 2 Kafka brokers with *42 partitions *on the topic
>
> With that configuration, no matter how I change some other configs (Please
> see list below), my topology is capping out at around 48k TPS.
> Please note that CPU usage, for the Supervisor nodes, is only around 20%
> and Network usage is only around 20Mbps for both the Kafka Brokers and
> Supervisor nodes, well below the network capacities.
>
> Now, I have increased the supervisor nodes from 6 to 12 and used a new
> topic with 82 partitions, thinking that scaling out could help the
> performance.
> So is this the new configuration:
>
> - *12 supervisor nodes *
> - 1 worker per node
> - 7 KafkaSpout executors per worker
> - 1 bolt (that does nothing) per worker
> - 0 Ackers
> - 2 Kafka brokers with *84 partitions* on the topic
>
> And I'm still getting around 48k TPS.
>
> Some other configs I played around with:
>
> - max.poll.records
> - max.spout.pending
> - processing.guarantee
> - offset.commit.period.ms
> - max.uncommitted.offsets
> - poll.timeout.ms
> - fetch.min.bytes
> - fetch.max.bytes
> - max.partition.fetch.bytes
> - receive.buffer.bytes
> - fetch.max.wait.ms
> - topology.executor.receive.buffer.size
> - topology.executor.send.buffer.size
> - topology.receiver.buffer.size
> - topology.transfer.buffer.size
>
>
> Am I missing something here? Why is the throughput not improving?
>
> Just to add, I have also done some performance isolation tests on both
> Kafka and Storm.
> On a distributed consumer using Spark and Kafka, I was able to get around
> 700K TPS (So we know that Kafka isn't the issue here).
> And also I could get around 400k TPS on a custom Storm topology with 1
> spout that generates random transactions and 1 bolt that does nothing.
> I feel like the numbers don't add up and the topology shouldn't be capping
> out at around 48K TPS.
> Your suggestions would be very much appreciated.
>
>
> Thanks,
> Bernard
>
>
Re: Kafka-Storm Performance Tests
Posted by Roshan Naik <ro...@yahoo.com>.
Just to narrow it down .. can u get rid of the bolt and see what numbers you get ?
Which Kafka spout are you using (new/old) ?
I assume you mean msgs when you say ‘transactions’. Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per worker ?
Sent from Yahoo Mail for iPhone
On Sunday, January 6, 2019, 11:29 PM, Bernard Quizon <be...@stellarloyalty.com> wrote:
Hi,
Before anything else, I'm using Apache Storm 1.2.1 and Apache Kafka 1.1.1.
On my initial tests, I have:
- 6 supervisor nodes
- 1 worker per node
- 7 KafkaSpout executors per worker
- 1 bolt (that does nothing) per worker
- 0 Ackers
- 2 Kafka brokers with 42 partitions on the topic
With that configuration, no matter how I change some other configs (Please see list below), my topology is capping out at around 48k TPS.Please note that CPU usage, for the Supervisor nodes, is only around 20% and Network usage is only around 20Mbps for both the Kafka Brokers and Supervisor nodes, well below the network capacities.
Now, I have increased the supervisor nodes from 6 to 12 and used a new topic with 82 partitions, thinking that scaling out could help the performance.So is this the new configuration:
- 12 supervisor nodes
- 1 worker per node
- 7 KafkaSpout executors per worker
- 1 bolt (that does nothing) per worker
- 0 Ackers
- 2 Kafka brokers with 84 partitions on the topic
And I'm still getting around 48k TPS.
Some other configs I played around with:
- max.poll.records
- max.spout.pending
- processing.guarantee
- offset.commit.period.ms
- max.uncommitted.offsets
- poll.timeout.ms
- fetch.min.bytes
- fetch.max.bytes
- max.partition.fetch.bytes
- receive.buffer.bytes
- fetch.max.wait.ms
- topology.executor.receive.buffer.size
- topology.executor.send.buffer.size
- topology.receiver.buffer.size
- topology.transfer.buffer.size
Am I missing something here? Why is the throughput not improving?
Just to add, I have also done some performance isolation tests on both Kafka and Storm.On a distributed consumer using Spark and Kafka, I was able to get around 700K TPS (So we know that Kafka isn't the issue here).And also I could get around 400k TPS on a custom Storm topology with 1 spout that generates random transactions and 1 bolt that does nothing.
I feel like the numbers don't add up and the topology shouldn't be capping out at around 48K TPS.Your suggestions would be very much appreciated.
Thanks,
Bernard