You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Idan Fridman <id...@gmail.com> on 2015/03/19 17:44:18 UTC
Not all spout tasks are active
I have topology using KafkaSpout
I defined my spouts configuration this way:
builder.setSpout("push-notification-reader", new
KafkaSpout(initKafkaSpoutConfig()),
4).setNumTasks(8).setMaxSpoutPending(50);
builder
public static SpoutConfig initKafkaSpoutConfig() {
BrokerHosts brokerHosts = new ZkHosts(BROKER_ZK_URL_CONSUME);
SpoutConfig kafkaConfig = new SpoutConfig(brokerHosts,
CONSUME_KAFKA_TOPIC_NAME, "", UUID.randomUUID().toString());
kafkaConfig.forceFromStart = false;
kafkaConfig.startOffsetTime = -1L;
kafkaConfig.scheme = new SchemeAsMultiScheme(new StringScheme());
return kafkaConfig;
}
I defined 3 workes:
config.setNumWorkers(3);
StormSubmitter.submitTopology(args[0], config, topology);
When you check in storm ui you see that only one instance is actually
receiving messages:
[image: תמונה מוטבעת 1]
[image: תמונה מוטבעת 2]
Any idea where the problem could be ?
thanks.
Re: Not all spout tasks are active
Posted by Idan Fridman <id...@gmail.com>.
8
2015-03-19 18:52 GMT+02:00 Adnan Karač <ad...@gmail.com>:
> How many partitions does your kafka topic have?
> ᐧ
>
> On Thu, Mar 19, 2015 at 5:44 PM, Idan Fridman <id...@gmail.com> wrote:
>
>> I have topology using KafkaSpout
>>
>> I defined my spouts configuration this way:
>>
>> builder.setSpout("push-notification-reader", new KafkaSpout(initKafkaSpoutConfig()), 4).setNumTasks(8).setMaxSpoutPending(50);
>> builder
>>
>> public static SpoutConfig initKafkaSpoutConfig() {
>> BrokerHosts brokerHosts = new ZkHosts(BROKER_ZK_URL_CONSUME);
>> SpoutConfig kafkaConfig = new SpoutConfig(brokerHosts, CONSUME_KAFKA_TOPIC_NAME, "", UUID.randomUUID().toString());
>> kafkaConfig.forceFromStart = false;
>> kafkaConfig.startOffsetTime = -1L;
>> kafkaConfig.scheme = new SchemeAsMultiScheme(new StringScheme());
>> return kafkaConfig;
>> }
>>
>>
>> I defined 3 workes:
>>
>> config.setNumWorkers(3);
>> StormSubmitter.submitTopology(args[0], config, topology);
>>
>> When you check in storm ui you see that only one instance is actually
>> receiving messages:
>>
>> [image: תמונה מוטבעת 1]
>>
>>
>> [image: תמונה מוטבעת 2]
>>
>> Any idea where the problem could be ?
>> thanks.
>>
>>
>>
>>
>
>
> --
> Adnan Karač
>
Re: Not all spout tasks are active
Posted by Adnan Karač <ad...@gmail.com>.
How many partitions does your kafka topic have?
ᐧ
On Thu, Mar 19, 2015 at 5:44 PM, Idan Fridman <id...@gmail.com> wrote:
> I have topology using KafkaSpout
>
> I defined my spouts configuration this way:
>
> builder.setSpout("push-notification-reader", new KafkaSpout(initKafkaSpoutConfig()), 4).setNumTasks(8).setMaxSpoutPending(50);
> builder
>
> public static SpoutConfig initKafkaSpoutConfig() {
> BrokerHosts brokerHosts = new ZkHosts(BROKER_ZK_URL_CONSUME);
> SpoutConfig kafkaConfig = new SpoutConfig(brokerHosts, CONSUME_KAFKA_TOPIC_NAME, "", UUID.randomUUID().toString());
> kafkaConfig.forceFromStart = false;
> kafkaConfig.startOffsetTime = -1L;
> kafkaConfig.scheme = new SchemeAsMultiScheme(new StringScheme());
> return kafkaConfig;
> }
>
>
> I defined 3 workes:
>
> config.setNumWorkers(3);
> StormSubmitter.submitTopology(args[0], config, topology);
>
> When you check in storm ui you see that only one instance is actually
> receiving messages:
>
> [image: תמונה מוטבעת 1]
>
>
> [image: תמונה מוטבעת 2]
>
> Any idea where the problem could be ?
> thanks.
>
>
>
>
--
Adnan Karač