Backpressure: Way to find out the pending messages at the Bolt

[Rajeev <ra...@gmail.com>]

Hi All,

I am using TCP communication to have 2 topologies communicate with each
other.
Below is an overview of the topologies:

Topology01:

Spout -> BoltA1 -> BoltA2 -> TCPProducerBolt (ex: port 5555 on a server).

Topology02( This receives data from the TCP port from the above topology.
Has a while loop to receive the data from TCP ):

TCPConsumerSpout -> BoltB1 -> BoltB2


I am afraid that if the tcp messages are too many for the BoltB1 to handle,
there may be know way to slow it down.
I need to know the number of pending messages in BoltB1 so that I can think
of somehow throttling TCPProducerBolt.

Kindly let me know if you have any suggestions.

Regards,
Rajeev.

Re: Backpressure: Way to find out the pending messages at the Bolt

[Roshan Naik <ro...@yahoo.com>]

Any reason it’s not a single topo ? Splitting it into two topos like this means you will need your own BP or just rely on TCP flow control. 
If you must split, You can consider routing via Kafka instead of TCP to mitigate data loss in case failures. 


Sent from Yahoo Mail for iPhone


On Thursday, November 8, 2018, 3:08 AM, Rajeev <ra...@gmail.com> wrote:

Hi All,
I am using TCP communication to have 2 topologies communicate with each other.Below is an overview of the topologies:

Topology01:
Spout -> BoltA1 -> BoltA2 -> TCPProducerBolt (ex: port 5555 on a server).
Topology02( This receives data from the TCP port from the above topology. Has a while loop to receive the data from TCP ):
TCPConsumerSpout -> BoltB1 -> BoltB2

I am afraid that if the tcp messages are too many for the BoltB1 to handle, there may be know way to slow it down.I need to know the number of pending messages in BoltB1 so that I can think of somehow throttling TCPProducerBolt.
Kindly let me know if you have any suggestions.
Regards,Rajeev.