How does the control flow in a Trident Topology work?

[Li Wang <wa...@gmail.com>]

Hi guys,

I am trying to understand the implementation of Trident. Through reading the code in TridentTopolgyBuilder.java, I understand that some Coordinator components, such as MasterBatchCoordinator and TridentSpoutCoordinator, are added to a user defined topology in TridentTopologyBuilder.createTopology(). I try to understand the control flow of those coordinators, but is seems to be very difficult to get the sense just from source code. Is there any document giving a high level of the control flow of the coordinator components in a Trident Topology?

Any help is highly appreciated. Thanks!

Sincerely,
Li Wang

Re: How does the control flow in a Trident Topology work?

[Manu Zhang <ow...@gmail.com>]

Hi Li,

Here is an old Chinese article on Trident
<http://www.cnblogs.com/hseagle/p/3490635.html> that explains the procedure
with some images. I think you read Chinese, right ?
The article is actually one in a series which may help you more to
understand Trident.

Cheers,
Manu Zhang

On Thu, Nov 3, 2016 at 10:55 PM Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:

> It is very confusing and i don't completely understand it all myself, but
> for the most part what happens, as far as I understand it, is that the
> master batch coordinator sends out begin batch messages to all of the spout
> coordinators and all the state instances saying batch X is about to begin.
> When it gets an ack back (saying everyone saw it and are ready for it) it
> will then send out a message saying emit batch X,  (all tuples within that
> batch are tied to this emit as the are processed).  If that comes back as
> success then it will send out a final message to commit batch X. (but only
> if batch X-1 has already been committed).  If any of these fail it will ask
> the spouts to replay the batch.
> - Bobby
>
> On Wednesday, November 2, 2016, 5:38:29 PM CDT, Li Wang <
> wangli1426@gmail.com> wrote:Hi guys,
>
> I am trying to understand the implementation of Trident. Through reading
> the code in TridentTopolgyBuilder.java, I understand that some Coordinator
> components, such as MasterBatchCoordinator and TridentSpoutCoordinator, are
> added to a user defined topology in
> TridentTopologyBuilder.createTopology(). I try to understand the control
> flow of those coordinators, but is seems to be very difficult to get the
> sense just from source code. Is there any document giving a high level of
> the control flow of the coordinator components in a Trident Topology?
>
> Any help is highly appreciated. Thanks!
>
> Sincerely,
> Li Wang

Re: How does the control flow in a Trident Topology work?

[Bobby Evans <ev...@yahoo-inc.com.INVALID>]

It is very confusing and i don't completely understand it all myself, but for the most part what happens, as far as I understand it, is that the master batch coordinator sends out begin batch messages to all of the spout coordinators and all the state instances saying batch X is about to begin.  When it gets an ack back (saying everyone saw it and are ready for it) it will then send out a message saying emit batch X,  (all tuples within that batch are tied to this emit as the are processed).  If that comes back as success then it will send out a final message to commit batch X. (but only if batch X-1 has already been committed).  If any of these fail it will ask the spouts to replay the batch.
- Bobby

On Wednesday, November 2, 2016, 5:38:29 PM CDT, Li Wang <wa...@gmail.com> wrote:Hi guys,

I am trying to understand the implementation of Trident. Through reading the code in TridentTopolgyBuilder.java, I understand that some Coordinator components, such as MasterBatchCoordinator and TridentSpoutCoordinator, are added to a user defined topology in TridentTopologyBuilder.createTopology(). I try to understand the control flow of those coordinators, but is seems to be very difficult to get the sense just from source code. Is there any document giving a high level of the control flow of the coordinator components in a Trident Topology?

Any help is highly appreciated. Thanks!

Sincerely,
Li Wang