You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Thomas Cristanis <th...@gmail.com> on 2017/02/07 12:53:21 UTC

Kafkaspout very slow

Hi,

I'm having a problem that I've tried a lot, and I can not find an answer.
I'm having slowness problem at the moment reading Kafka's data with
Trident. For example in about 10 minutes were issued in the spout less than
120 thousand messages.

BrokerHosts hosts = new ZkHosts(Configuration.ZOOKEEPER_CONNECTION_STRING);

TridentKafkaConfig spoutConf = new TridentKafkaConfig(hosts, topicName);

spoutConf.scheme = new SchemeAsMultiScheme(new StringScheme());

spoutConf.ignoreZkOffsets = true;

spoutConf.startOffsetTime = -1;

spoutConf.bufferSizeBytes = 1024 * 1024 * 4;

spoutConf.fetchSizeBytes = 1024 * 1024 * 4;


OpaqueTridentKafkaSpout spout = new OpaqueTridentKafkaSpout(spoutConf);

Stream stream = topologyTrident.newStream("lines",
spout).parallelismHint(8);



--
Thomas Cristanis

Re: Kafkaspout very slow

Posted by Thomas Cristanis <th...@gmail.com>.
Yeah, sure.
I'm running on a cluster; there are four machines (32 GB each): one machine
with Apache Kafka, one running the nimbus and another two running each one
the supervisor. Using these same machines, I have another topology using
Storm Core, and I have a performance of more than 100k emitted per second.


--
Thomas Cristanis

2017-02-07 9:58 GMT-03:00 Andrew Xor <an...@gmail.com>:

> Hello Thomas,
>
>  Can you give us a bit more information about how things are being run?
> Does this happen on remote clusters? Local? Single machine? Kafka is run on
> the same cluster? There are a lot of things that might affect spout
> performance in general that unfortunately we cannot discern by a few lines
> of code... Please give us information on the architecture you are using, so
> we can help you.
>
> Warm regards,
>
> A.
>
> On Tue, Feb 7, 2017 at 12:53 PM, Thomas Cristanis <
> thomascristanis@gmail.com> wrote:
>
>> Hi,
>>
>> I'm having a problem that I've tried a lot, and I can not find an answer.
>> I'm having slowness problem at the moment reading Kafka's data with
>> Trident. For example in about 10 minutes were issued in the spout less than
>> 120 thousand messages.
>>
>> BrokerHosts hosts = new ZkHosts(Configuration.ZOOKEEPER_CONNECTION_STRING
>> );
>>
>> TridentKafkaConfig spoutConf = new TridentKafkaConfig(hosts, topicName);
>>
>> spoutConf.scheme = new SchemeAsMultiScheme(new StringScheme());
>>
>> spoutConf.ignoreZkOffsets = true;
>>
>> spoutConf.startOffsetTime = -1;
>>
>> spoutConf.bufferSizeBytes = 1024 * 1024 * 4;
>>
>> spoutConf.fetchSizeBytes = 1024 * 1024 * 4;
>>
>>
>> OpaqueTridentKafkaSpout spout = new OpaqueTridentKafkaSpout(spoutConf);
>>
>> Stream stream = topologyTrident.newStream("lines",
>> spout).parallelismHint(8);
>>
>>
>>
>> --
>> Thomas Cristanis
>>
>
>

Re: Kafkaspout very slow

Posted by Andrew Xor <an...@gmail.com>.
Hello Thomas,

 Can you give us a bit more information about how things are being run?
Does this happen on remote clusters? Local? Single machine? Kafka is run on
the same cluster? There are a lot of things that might affect spout
performance in general that unfortunately we cannot discern by a few lines
of code... Please give us information on the architecture you are using, so
we can help you.

Warm regards,

A.

On Tue, Feb 7, 2017 at 12:53 PM, Thomas Cristanis <thomascristanis@gmail.com
> wrote:

> Hi,
>
> I'm having a problem that I've tried a lot, and I can not find an answer.
> I'm having slowness problem at the moment reading Kafka's data with
> Trident. For example in about 10 minutes were issued in the spout less than
> 120 thousand messages.
>
> BrokerHosts hosts = new ZkHosts(Configuration.ZOOKEEPER_CONNECTION_STRING
> );
>
> TridentKafkaConfig spoutConf = new TridentKafkaConfig(hosts, topicName);
>
> spoutConf.scheme = new SchemeAsMultiScheme(new StringScheme());
>
> spoutConf.ignoreZkOffsets = true;
>
> spoutConf.startOffsetTime = -1;
>
> spoutConf.bufferSizeBytes = 1024 * 1024 * 4;
>
> spoutConf.fetchSizeBytes = 1024 * 1024 * 4;
>
>
> OpaqueTridentKafkaSpout spout = new OpaqueTridentKafkaSpout(spoutConf);
>
> Stream stream = topologyTrident.newStream("lines",
> spout).parallelismHint(8);
>
>
>
> --
> Thomas Cristanis
>