You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Hyunseok Chang <hy...@gmail.com> on 2016/08/30 14:22:23 UTC
Support for dynamic topology
Hi,
I'd like to know more about Apex support for dynamic topology.
From my readings on Apex, I understand we can add additional parallel tasks
for each operator and change data partitioning among them dynamically at
run time (so-called data partitioning and unification features).
My question is can we change the "logical" DAG at run time?
Let's say my logical DAG is a chain of three operators X, Y & Z (i.e., X ->
Y -> Z). Now at run time I want to replace operator Y with operator P,
such that the new logical DAG would look like X -> P -> Z.
Is it something I can do with Apex?
Thanks!
-hs
Re: Support for dynamic topology
Posted by Amol Kekre <am...@datatorrent.com>.
Hyunseok,
From the DAG it looks like single port. In case that is true, one way to
get around this is to introduce a dummy pass-through operator during design
of the app itself. This pass through operator shoudl have two input ports
and one output port. One of the port is getting data from Y, other is open
during launch time. During run time try inserting new operator P and
connect it to this open port, and then disconned Y. A little tricky, but
see if it works. The danger here is that the connects and disconnects
should be triggered on the same window boundary, else you will have a
window with both old and new operators are active.
Thks
Amol
On Tue, Aug 30, 2016 at 1:57 PM, Thomas Weise <th...@gmail.com>
wrote:
> Hi,
>
> This depends on the operator Z. If it has multiple input ports and those
> are optional, you can add P, then connect P to Z (and X), then remove Y.
>
> If Z has a single port, then Z and everything downstream would need to be
> removed or else the change won't result in a valid DAG and won't be
> accepted.
>
> Thomas
>
>
> On Tue, Aug 30, 2016 at 1:48 PM, Hyunseok Chang <hy...@gmail.com>
> wrote:
>
>> So if I replace Y in pipeline X -> Y -> Z -> U -> W -> V with P, then
>> what I would have is X -> P -> Z' -> U' -> W' -> V' ? Where Z', U', W'
>> and V' are new operator instances that need to be deployed along with P.
>>
>> Is my understanding correct? If so is there any reason why we cannot
>> re-use existing operators downstream?
>>
>> -hs
>>
>>
>> On Tue, Aug 30, 2016 at 2:46 PM, Amol Kekre <am...@datatorrent.com> wrote:
>>
>>>
>>> Hyunseok,
>>> The new route in the pipeline will have a new Z operator. If you want to
>>> use the old Z operator (state?) then things get tricky. Do confirm that you
>>> do not plan to use old Z operator.
>>>
>>> Thks,
>>> Amol
>>>
>>>
>>> On Tue, Aug 30, 2016 at 11:02 AM, Sandesh Hegde <sandesh@datatorrent.com
>>> > wrote:
>>>
>>>> Hello hs,
>>>>
>>>> Yes, you can change the topology from the Apex CLI.
>>>>
>>>> One possible sequence of commands for your scenario is described below,
>>>>
>>>> connect appid
>>>> begin-logical-plan-change
>>>> create-operator ....
>>>> add-stream-sink ... ( for the input of P )
>>>> add-stream-sink ... ( for the output of P )
>>>> remove-operator ...
>>>> submit
>>>>
>>>> Note: All the required operators needs to be in the package.
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Aug 30, 2016 at 7:22 AM Hyunseok Chang <
>>>> hyunseok.chang@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'd like to know more about Apex support for dynamic topology.
>>>>>
>>>>> From my readings on Apex, I understand we can add additional parallel
>>>>> tasks for each operator and change data partitioning among them dynamically
>>>>> at run time (so-called data partitioning and unification features).
>>>>>
>>>>> My question is can we change the "logical" DAG at run time?
>>>>>
>>>>> Let's say my logical DAG is a chain of three operators X, Y & Z (i.e.,
>>>>> X -> Y -> Z). Now at run time I want to replace operator Y with operator
>>>>> P, such that the new logical DAG would look like X -> P -> Z.
>>>>>
>>>>> Is it something I can do with Apex?
>>>>>
>>>>> Thanks!
>>>>> -hs
>>>>>
>>>>>
>>>
>>
>
Re: Support for dynamic topology
Posted by Thomas Weise <th...@gmail.com>.
Hi,
This depends on the operator Z. If it has multiple input ports and those
are optional, you can add P, then connect P to Z (and X), then remove Y.
If Z has a single port, then Z and everything downstream would need to be
removed or else the change won't result in a valid DAG and won't be
accepted.
Thomas
On Tue, Aug 30, 2016 at 1:48 PM, Hyunseok Chang <hy...@gmail.com>
wrote:
> So if I replace Y in pipeline X -> Y -> Z -> U -> W -> V with P, then what
> I would have is X -> P -> Z' -> U' -> W' -> V' ? Where Z', U', W' and V'
> are new operator instances that need to be deployed along with P.
>
> Is my understanding correct? If so is there any reason why we cannot
> re-use existing operators downstream?
>
> -hs
>
>
> On Tue, Aug 30, 2016 at 2:46 PM, Amol Kekre <am...@datatorrent.com> wrote:
>
>>
>> Hyunseok,
>> The new route in the pipeline will have a new Z operator. If you want to
>> use the old Z operator (state?) then things get tricky. Do confirm that you
>> do not plan to use old Z operator.
>>
>> Thks,
>> Amol
>>
>>
>> On Tue, Aug 30, 2016 at 11:02 AM, Sandesh Hegde <sa...@datatorrent.com>
>> wrote:
>>
>>> Hello hs,
>>>
>>> Yes, you can change the topology from the Apex CLI.
>>>
>>> One possible sequence of commands for your scenario is described below,
>>>
>>> connect appid
>>> begin-logical-plan-change
>>> create-operator ....
>>> add-stream-sink ... ( for the input of P )
>>> add-stream-sink ... ( for the output of P )
>>> remove-operator ...
>>> submit
>>>
>>> Note: All the required operators needs to be in the package.
>>>
>>> Thanks
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Aug 30, 2016 at 7:22 AM Hyunseok Chang <hy...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'd like to know more about Apex support for dynamic topology.
>>>>
>>>> From my readings on Apex, I understand we can add additional parallel
>>>> tasks for each operator and change data partitioning among them dynamically
>>>> at run time (so-called data partitioning and unification features).
>>>>
>>>> My question is can we change the "logical" DAG at run time?
>>>>
>>>> Let's say my logical DAG is a chain of three operators X, Y & Z (i.e.,
>>>> X -> Y -> Z). Now at run time I want to replace operator Y with operator
>>>> P, such that the new logical DAG would look like X -> P -> Z.
>>>>
>>>> Is it something I can do with Apex?
>>>>
>>>> Thanks!
>>>> -hs
>>>>
>>>>
>>
>
Re: Support for dynamic topology
Posted by Hyunseok Chang <hy...@gmail.com>.
So if I replace Y in pipeline X -> Y -> Z -> U -> W -> V with P, then what
I would have is X -> P -> Z' -> U' -> W' -> V' ? Where Z', U', W' and V'
are new operator instances that need to be deployed along with P.
Is my understanding correct? If so is there any reason why we cannot
re-use existing operators downstream?
-hs
On Tue, Aug 30, 2016 at 2:46 PM, Amol Kekre <am...@datatorrent.com> wrote:
>
> Hyunseok,
> The new route in the pipeline will have a new Z operator. If you want to
> use the old Z operator (state?) then things get tricky. Do confirm that you
> do not plan to use old Z operator.
>
> Thks,
> Amol
>
>
> On Tue, Aug 30, 2016 at 11:02 AM, Sandesh Hegde <sa...@datatorrent.com>
> wrote:
>
>> Hello hs,
>>
>> Yes, you can change the topology from the Apex CLI.
>>
>> One possible sequence of commands for your scenario is described below,
>>
>> connect appid
>> begin-logical-plan-change
>> create-operator ....
>> add-stream-sink ... ( for the input of P )
>> add-stream-sink ... ( for the output of P )
>> remove-operator ...
>> submit
>>
>> Note: All the required operators needs to be in the package.
>>
>> Thanks
>>
>>
>>
>>
>>
>> On Tue, Aug 30, 2016 at 7:22 AM Hyunseok Chang <hy...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I'd like to know more about Apex support for dynamic topology.
>>>
>>> From my readings on Apex, I understand we can add additional parallel
>>> tasks for each operator and change data partitioning among them dynamically
>>> at run time (so-called data partitioning and unification features).
>>>
>>> My question is can we change the "logical" DAG at run time?
>>>
>>> Let's say my logical DAG is a chain of three operators X, Y & Z (i.e., X
>>> -> Y -> Z). Now at run time I want to replace operator Y with operator P,
>>> such that the new logical DAG would look like X -> P -> Z.
>>>
>>> Is it something I can do with Apex?
>>>
>>> Thanks!
>>> -hs
>>>
>>>
>
Re: Support for dynamic topology
Posted by Amol Kekre <am...@datatorrent.com>.
Hyunseok,
The new route in the pipeline will have a new Z operator. If you want to
use the old Z operator (state?) then things get tricky. Do confirm that you
do not plan to use old Z operator.
Thks,
Amol
On Tue, Aug 30, 2016 at 11:02 AM, Sandesh Hegde <sa...@datatorrent.com>
wrote:
> Hello hs,
>
> Yes, you can change the topology from the Apex CLI.
>
> One possible sequence of commands for your scenario is described below,
>
> connect appid
> begin-logical-plan-change
> create-operator ....
> add-stream-sink ... ( for the input of P )
> add-stream-sink ... ( for the output of P )
> remove-operator ...
> submit
>
> Note: All the required operators needs to be in the package.
>
> Thanks
>
>
>
>
>
> On Tue, Aug 30, 2016 at 7:22 AM Hyunseok Chang <hy...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I'd like to know more about Apex support for dynamic topology.
>>
>> From my readings on Apex, I understand we can add additional parallel
>> tasks for each operator and change data partitioning among them dynamically
>> at run time (so-called data partitioning and unification features).
>>
>> My question is can we change the "logical" DAG at run time?
>>
>> Let's say my logical DAG is a chain of three operators X, Y & Z (i.e., X
>> -> Y -> Z). Now at run time I want to replace operator Y with operator P,
>> such that the new logical DAG would look like X -> P -> Z.
>>
>> Is it something I can do with Apex?
>>
>> Thanks!
>> -hs
>>
>>
Re: Support for dynamic topology
Posted by Sandesh Hegde <sa...@datatorrent.com>.
Hello hs,
Yes, you can change the topology from the Apex CLI.
One possible sequence of commands for your scenario is described below,
connect appid
begin-logical-plan-change
create-operator ....
add-stream-sink ... ( for the input of P )
add-stream-sink ... ( for the output of P )
remove-operator ...
submit
Note: All the required operators needs to be in the package.
Thanks
On Tue, Aug 30, 2016 at 7:22 AM Hyunseok Chang <hy...@gmail.com>
wrote:
> Hi,
>
> I'd like to know more about Apex support for dynamic topology.
>
> From my readings on Apex, I understand we can add additional parallel
> tasks for each operator and change data partitioning among them dynamically
> at run time (so-called data partitioning and unification features).
>
> My question is can we change the "logical" DAG at run time?
>
> Let's say my logical DAG is a chain of three operators X, Y & Z (i.e., X
> -> Y -> Z). Now at run time I want to replace operator Y with operator P,
> such that the new logical DAG would look like X -> P -> Z.
>
> Is it something I can do with Apex?
>
> Thanks!
> -hs
>
>