You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "Rick Kellogg (JIRA)" <ji...@apache.org> on 2015/09/26 04:37:04 UTC
[jira] [Updated] (STORM-418) using storm kafka spout doesn't scale
when increasing topic partition
[ https://issues.apache.org/jira/browse/STORM-418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rick Kellogg updated STORM-418:
-------------------------------
Component/s: storm-kafka
> using storm kafka spout doesn't scale when increasing topic partition
> ---------------------------------------------------------------------
>
> Key: STORM-418
> URL: https://issues.apache.org/jira/browse/STORM-418
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka
> Affects Versions: 0.9.2-incubating
> Environment: EC2 c3.8x large instances for supervisor and kafka-0.8.1.1 running Amazon Linux 64 bit version OS.
> Kafka is using 4 EBS drives of 4000 IOPS of size 150 GB each.
> Reporter: Kumar Gaurav
> Labels: performance
>
> We are running a topology of kafka spout + single bolt on a single supervisor machine.
> Bolt functionality is limited to log parsed data from spout only.
> When Trying to consume same type of data by different topics using varying topic partitions topology throughput remain same in order of 70000 msg/sec.
> However, when multiple spout instances are used in same topology , we were able to increase topology throughput to more than 100,000 msg/sec.
> We have tried to increase kafka topic partition from 8 to 200, however topology throughput remains same for sinlge spout instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)