Re: MasterBatchCoordinator has become the bottleneck in my trident topology, can the number of MasterBatchCoordinator task larger than 1?

[Satish Duggana <sa...@gmail.com>]

AFAIK,  there can be only one master-batch-coordinator for each batch
group. You should look at increasing parallelism of trident operations to
get better latencies.

Thanks,
Satish.

On Wed, Jul 13, 2016 at 12:26 PM, hong mao <ma...@gmail.com> wrote:

> I get the resaon, there is only one MasterBatchCoordinator for each
> trident spout, so setting the thrid parameter (parallelism) takes no
> effect.
>
>
> 2016-07-12 0:08 GMT+08:00 hong mao <ma...@gmail.com>:
>
>> Hi all,
>>      We are using trident topology to pull messages from kafka and store
>> into hbase using partitionPersist. Recently we meet with a following
>> situation, the latency of the topology is increasing, as is shown in
>> stormUI, large portion of the latency is taken by $mastercoord-bg0
>> <http://10.139.19.81:8080/component.html?id=%24mastercoord-bg0&topology_id=biztracer-product-73-1468239144> ,
>> which corresponds to MasterBatchCoordinator, and its task parallelism is 1.
>> By checking source code TridentTopologyBuilder.java, I find that there is
>> no way to configure the task parallelism of MasterBatchCoordinator (MBC).
>>
>> for(String batch: batchesToCommitIds.keySet()) {
>>> List<String> commitIds = batchesToCommitIds.get(batch);
>>> builder.setSpout(masterCoordinator(batch), new
>>> MasterBatchCoordinator(commitIds, batchesToSpouts.get(batch)));
>>> }
>>>
>>
>> Should I just write a new Builder which overrides TridentTopologyBuilder
>> so that I can increase the task parallelism of MasterBatchCoordinator and
>> imporve the throughput of MBC ?
>>
>> If it is ok, may I open an issue to solve this problem ?
>>
>>
>> Thanks a lot!
>> ​
>>
>
>