You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Tushar Gosavi <tu...@datatorrent.com> on 2015/08/31 19:35:00 UTC
Modules support in Apex
Hi All,
We are working on adding support for Modules in Apex. A module is a group
of operators that will have their own existence and will ease the way we
are currently defining an application.
A module is defined by:
- list of operators.
- list of input and output ports.
- set of properties for the module.
- set of attributes for the module.
Details on the proposed work is given at:
https://malhar.atlassian.net/browse/APEX-55 . The work is also related to
https://malhar.atlassian.net/browse/APEX-3
Regards,
-Tushar.
Re: Modules support in Apex
Posted by Vlad Rozov <v....@datatorrent.com>.
There is https://malhar.atlassian.net/browse/APEX-23 open by Chetan for
replacement of dynamic operators aka modules at runtime. I closed it as
a duplicate in favor of discussion that already going on APEX-3.
Thank you,
Vlad
On 9/4/15 01:07, Atri Sharma wrote:
> Please find JIRA:
>
> https://malhar.atlassian.net/browse/APEX-95
>
> On Fri, Sep 4, 2015 at 2:43 AM, Amol Kekre <am...@datatorrent.com> wrote:
>
>> Atri,
>> Dynamic changes were in the plans. We missed putting it in open source
>> jira. Please open a jira in APEX for dynamic changes to DAG.
>>
>> Thks,
>> Amol
>>
>> On Thu, Sep 3, 2015 at 1:09 PM, Atri Sharma <at...@gmail.com> wrote:
>>
>>>> On Thu, Sep 3, 2015 at 11:55 AM, Atri Sharma <at...@gmail.com>
>>> wrote:
>>>>> Thanks for the detailed explanation!
>>>>>
>>>>> I have few small questions. Please find them inline.
>>>>>
>>>>> On 3 Sep 2015 21:03, "Amol Kekre" <am...@datatorrent.com> wrote:
>>>>>
>>>>>> Design time -> design, try, iterate => Re-use of IP saves a lot of
>>> time
>>>>>> here. Operators help as leaf level IP, Modules help as higher level
>>> IP
>>>>> that
>>>>>> can be made by combining previously tested leaf level IP.
>>> Additionally
>>>>> they
>>>>>> should be (a) correct by construction due to leveraging other
>>> operators
>>>>>> (b) allow properties, attributes to make re-use IP log more
>> powerful.
>>>>>> Modules do not prohibit Apex engine from optimizing DAG during
>> launch
>>> (or
>>>>>> even run) time, all the while enabling re-use at higher level. In
>>> fact
>>>>> they
>>>>>> may aid optimization by giving specific hints at a sub-DAG level.
>>> Same as
>>>>>> functions, templates, etc. in C++. This phase is mainly human time.
>>> This
>>>>>> results in a "logical plan" of App DAG
>>>>>>
>>>>> So essentially a logical plan is the one manually defined by the
>> users
>>>>> using operators and modules, correct?
>>>>>
>>>> Yes.
>>> Thanks!
>>>
>>>>> (I understand logical plan is generated from JSON files, just want to
>>>>> clarify if the JSON files are the one defined by user to define the
>>> DAG).
>>>>>> Launch time -> Usually a one time cost. This is always completely
>>>>>> automated in any compiler (aka compile time). So stuffing more here
>>> moves
>>>>>> us from human time to computer time. Module moves a lot more work
>>> from
>>>>>> human time to computer time as compared to leaf level operator.
>> This
>>>>>> results in physical plan of App DAG
>>>>> If I understand correctly, this is where we expand Modules I. E.
>> call
>>>>> Module's populateDAG method, correct?
>>>>>
>>>>>
>>>> Yes
>>> Thanks!
>>>
>>>>
>>>>>> Run time -> Actual app run time. At this stage the flattening has
>>>>> already
>>>>>> happened. Apex apps are to run for-ever or for a long time. These
>> are
>>> not
>>>>> 1
>>>>>> min apps. So an extra 10 seconds during launch time in a big data
>>> project
>>>>>> is amortized away over a days, months, or at least a few hours.
>>>>> Got that, thanks!
>>>>>
>>>>>> Operators do that at leaf level, modules enable true distributed
>>>>> execution.
>>>>>
>>>>> Is the distributed nature due to reuse?
>>>>>
>>>>>
>>>> Distributed nature is because Apex engine can now take this dag (called
>>>> physical plan), get resources from RM, and create an execution plan
>>> (where
>>>> each operator runs). Apex engine then distributes it. By flattening the
>>>> module in logical plan to physical plan phase, we leverage Apex
>> engine's
>>>> in-built distributive execution. The logical plan -> physical plan ->
>>>> execution plan is very common across a lot of distributed engines.
>>>>
>>> Got that, thanks.
>>>>>> I am hoping to see this thesis proved by making Calcite integration
>>>>> easier.
>>>>>
>>>>> I totally undrstand what Modules are meant to be and find them
>>> fascinating.
>>>>> I have a couple of questions around their usage please:
>>>>>
>>>>> 1) Is there any difference between a user using modules for reusing
>>> static
>>>>> DAGs vs expansion of Modules at runtime?
>>>>> 2) After Modules are introduced, will APEX-3's objective be to have
>>> dynamic
>>>>> flattening of Modules at runtime?
>>>>>
>>>>>
>>>> Apex-3 simply enables launch time ability, i.e static DAG. The
>> objective
>>> is
>>>> to get dynamic DAG changes at module level. Currently they are at
>>> operator
>>>> level. Their is no jira for dynamic module DAG changes akaik. But we
>> did
>>>> talk about it before Apex incubated.
>>> I think my attack plans have been around dynamic DAG changes since that
>> is
>>> what I understood from our discussion offline earlier. I shall continue
>>> working with Vlad in APEX 3 and shall open new JIRA for dynamic changes
>> if
>>> it is ok.
>>>>
>>>>>> On Thu, Sep 3, 2015 at 1:12 AM, Atri Sharma <at...@apache.org>
>> wrote:
>>>>>>> Amol.
>>>>>>>
>>>>>>> For my understanding, when you mention launch time/code
>> generation
>>>>> time,
>>>>>>> are you referring to generation of physical plan, please?
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Atri
>>>>>>>
>>>>>>> On Thu, Sep 3, 2015 at 12:48 PM, Amol Kekre <
>> amol@datatorrent.com>
>>>>> wrote:
>>>>>>>> Atri,
>>>>>>>> For a lot of operations module should be treated as a black
>> box.
>>> It
>>>>> is
>>>>>>> just
>>>>>>>> another reusable IP. The flattening should happen at launch
>> time.
>>>>>>>> If we think of Apex as a compiler, then all the compile time
>>> checks
>>>>>>> (ports
>>>>>>>> connectivity, matching types/schema, properties, attributes,
>> ...)
>>> are
>>>>> as
>>>>>>>> applicable to modules as to operators. At launch time (aka code
>>>>>>> generation
>>>>>>>> time) module gets flattened. Webservice should still enable
>>> access
>>>>> via
>>>>>>>> module scope on a running app.
>>>>>>>>
>>>>>>>> Thks,
>>>>>>>> Amol
>>>>>>>>
>
>
Re: Modules support in Apex
Posted by Atri Sharma <at...@gmail.com>.
Please find JIRA:
https://malhar.atlassian.net/browse/APEX-95
On Fri, Sep 4, 2015 at 2:43 AM, Amol Kekre <am...@datatorrent.com> wrote:
> Atri,
> Dynamic changes were in the plans. We missed putting it in open source
> jira. Please open a jira in APEX for dynamic changes to DAG.
>
> Thks,
> Amol
>
> On Thu, Sep 3, 2015 at 1:09 PM, Atri Sharma <at...@gmail.com> wrote:
>
> > > On Thu, Sep 3, 2015 at 11:55 AM, Atri Sharma <at...@gmail.com>
> > wrote:
> > >
> > > > Thanks for the detailed explanation!
> > > >
> > > > I have few small questions. Please find them inline.
> > > >
> > > > On 3 Sep 2015 21:03, "Amol Kekre" <am...@datatorrent.com> wrote:
> > > >
> > > > > Design time -> design, try, iterate => Re-use of IP saves a lot of
> > time
> > > > > here. Operators help as leaf level IP, Modules help as higher level
> > IP
> > > > that
> > > > > can be made by combining previously tested leaf level IP.
> > Additionally
> > > > they
> > > > > should be (a) correct by construction due to leveraging other
> > operators
> > > > > (b) allow properties, attributes to make re-use IP log more
> powerful.
> > > > > Modules do not prohibit Apex engine from optimizing DAG during
> launch
> > (or
> > > > > even run) time, all the while enabling re-use at higher level. In
> > fact
> > > > they
> > > > > may aid optimization by giving specific hints at a sub-DAG level.
> > Same as
> > > > > functions, templates, etc. in C++. This phase is mainly human time.
> > This
> > > > > results in a "logical plan" of App DAG
> > > > >
> > > >
> > > > So essentially a logical plan is the one manually defined by the
> users
> > > > using operators and modules, correct?
> > > >
> > >
> > > Yes.
> >
> > Thanks!
> >
> > >
> > > >
> > > > (I understand logical plan is generated from JSON files, just want to
> > > > clarify if the JSON files are the one defined by user to define the
> > DAG).
> > > >
> > > > > Launch time -> Usually a one time cost. This is always completely
> > > > > automated in any compiler (aka compile time). So stuffing more here
> > moves
> > > > > us from human time to computer time. Module moves a lot more work
> > from
> > > > > human time to computer time as compared to leaf level operator.
> This
> > > > > results in physical plan of App DAG
> > > >
> > > > If I understand correctly, this is where we expand Modules I. E.
> call
> > > > Module's populateDAG method, correct?
> > > >
> > > >
> > > Yes
> >
> > Thanks!
> >
> > >
> > >
> > > > > Run time -> Actual app run time. At this stage the flattening has
> > > > already
> > > > > happened. Apex apps are to run for-ever or for a long time. These
> are
> > not
> > > > 1
> > > > > min apps. So an extra 10 seconds during launch time in a big data
> > project
> > > > > is amortized away over a days, months, or at least a few hours.
> > > >
> > > > Got that, thanks!
> > > >
> > > > > Operators do that at leaf level, modules enable true distributed
> > > > execution.
> > > >
> > > > Is the distributed nature due to reuse?
> > > >
> > > >
> > > Distributed nature is because Apex engine can now take this dag (called
> > > physical plan), get resources from RM, and create an execution plan
> > (where
> > > each operator runs). Apex engine then distributes it. By flattening the
> > > module in logical plan to physical plan phase, we leverage Apex
> engine's
> > > in-built distributive execution. The logical plan -> physical plan ->
> > > execution plan is very common across a lot of distributed engines.
> > >
> > Got that, thanks.
> > >
> > > > > I am hoping to see this thesis proved by making Calcite integration
> > > > easier.
> > > >
> > > > I totally undrstand what Modules are meant to be and find them
> > fascinating.
> > > > I have a couple of questions around their usage please:
> > > >
> > > > 1) Is there any difference between a user using modules for reusing
> > static
> > > > DAGs vs expansion of Modules at runtime?
> > > > 2) After Modules are introduced, will APEX-3's objective be to have
> > dynamic
> > > > flattening of Modules at runtime?
> > > >
> > > >
> > > Apex-3 simply enables launch time ability, i.e static DAG. The
> objective
> > is
> > > to get dynamic DAG changes at module level. Currently they are at
> > operator
> > > level. Their is no jira for dynamic module DAG changes akaik. But we
> did
> > > talk about it before Apex incubated.
> >
> > I think my attack plans have been around dynamic DAG changes since that
> is
> > what I understood from our discussion offline earlier. I shall continue
> > working with Vlad in APEX 3 and shall open new JIRA for dynamic changes
> if
> > it is ok.
> > >
> > >
> > > > >
> > > > > On Thu, Sep 3, 2015 at 1:12 AM, Atri Sharma <at...@apache.org>
> wrote:
> > > > >
> > > > > > Amol.
> > > > > >
> > > > > > For my understanding, when you mention launch time/code
> generation
> > > > time,
> > > > > > are you referring to generation of physical plan, please?
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Atri
> > > > > >
> > > > > > On Thu, Sep 3, 2015 at 12:48 PM, Amol Kekre <
> amol@datatorrent.com>
> > > > wrote:
> > > > > >
> > > > > > > Atri,
> > > > > > > For a lot of operations module should be treated as a black
> box.
> > It
> > > > is
> > > > > > just
> > > > > > > another reusable IP. The flattening should happen at launch
> time.
> > > > > > >
> > > > > > > If we think of Apex as a compiler, then all the compile time
> > checks
> > > > > > (ports
> > > > > > > connectivity, matching types/schema, properties, attributes,
> ...)
> > are
> > > > as
> > > > > > > applicable to modules as to operators. At launch time (aka code
> > > > > > generation
> > > > > > > time) module gets flattened. Webservice should still enable
> > access
> > > > via
> > > > > > > module scope on a running app.
> > > > > > >
> > > > > > > Thks,
> > > > > > > Amol
> > > > > > >
> > > > >
> > > >
> >
>
--
Regards,
Atri
*l'apprenant*
Re: Modules support in Apex
Posted by Amol Kekre <am...@datatorrent.com>.
Atri,
Dynamic changes were in the plans. We missed putting it in open source
jira. Please open a jira in APEX for dynamic changes to DAG.
Thks,
Amol
On Thu, Sep 3, 2015 at 1:09 PM, Atri Sharma <at...@gmail.com> wrote:
> > On Thu, Sep 3, 2015 at 11:55 AM, Atri Sharma <at...@gmail.com>
> wrote:
> >
> > > Thanks for the detailed explanation!
> > >
> > > I have few small questions. Please find them inline.
> > >
> > > On 3 Sep 2015 21:03, "Amol Kekre" <am...@datatorrent.com> wrote:
> > >
> > > > Design time -> design, try, iterate => Re-use of IP saves a lot of
> time
> > > > here. Operators help as leaf level IP, Modules help as higher level
> IP
> > > that
> > > > can be made by combining previously tested leaf level IP.
> Additionally
> > > they
> > > > should be (a) correct by construction due to leveraging other
> operators
> > > > (b) allow properties, attributes to make re-use IP log more powerful.
> > > > Modules do not prohibit Apex engine from optimizing DAG during launch
> (or
> > > > even run) time, all the while enabling re-use at higher level. In
> fact
> > > they
> > > > may aid optimization by giving specific hints at a sub-DAG level.
> Same as
> > > > functions, templates, etc. in C++. This phase is mainly human time.
> This
> > > > results in a "logical plan" of App DAG
> > > >
> > >
> > > So essentially a logical plan is the one manually defined by the users
> > > using operators and modules, correct?
> > >
> >
> > Yes.
>
> Thanks!
>
> >
> > >
> > > (I understand logical plan is generated from JSON files, just want to
> > > clarify if the JSON files are the one defined by user to define the
> DAG).
> > >
> > > > Launch time -> Usually a one time cost. This is always completely
> > > > automated in any compiler (aka compile time). So stuffing more here
> moves
> > > > us from human time to computer time. Module moves a lot more work
> from
> > > > human time to computer time as compared to leaf level operator. This
> > > > results in physical plan of App DAG
> > >
> > > If I understand correctly, this is where we expand Modules I. E. call
> > > Module's populateDAG method, correct?
> > >
> > >
> > Yes
>
> Thanks!
>
> >
> >
> > > > Run time -> Actual app run time. At this stage the flattening has
> > > already
> > > > happened. Apex apps are to run for-ever or for a long time. These are
> not
> > > 1
> > > > min apps. So an extra 10 seconds during launch time in a big data
> project
> > > > is amortized away over a days, months, or at least a few hours.
> > >
> > > Got that, thanks!
> > >
> > > > Operators do that at leaf level, modules enable true distributed
> > > execution.
> > >
> > > Is the distributed nature due to reuse?
> > >
> > >
> > Distributed nature is because Apex engine can now take this dag (called
> > physical plan), get resources from RM, and create an execution plan
> (where
> > each operator runs). Apex engine then distributes it. By flattening the
> > module in logical plan to physical plan phase, we leverage Apex engine's
> > in-built distributive execution. The logical plan -> physical plan ->
> > execution plan is very common across a lot of distributed engines.
> >
> Got that, thanks.
> >
> > > > I am hoping to see this thesis proved by making Calcite integration
> > > easier.
> > >
> > > I totally undrstand what Modules are meant to be and find them
> fascinating.
> > > I have a couple of questions around their usage please:
> > >
> > > 1) Is there any difference between a user using modules for reusing
> static
> > > DAGs vs expansion of Modules at runtime?
> > > 2) After Modules are introduced, will APEX-3's objective be to have
> dynamic
> > > flattening of Modules at runtime?
> > >
> > >
> > Apex-3 simply enables launch time ability, i.e static DAG. The objective
> is
> > to get dynamic DAG changes at module level. Currently they are at
> operator
> > level. Their is no jira for dynamic module DAG changes akaik. But we did
> > talk about it before Apex incubated.
>
> I think my attack plans have been around dynamic DAG changes since that is
> what I understood from our discussion offline earlier. I shall continue
> working with Vlad in APEX 3 and shall open new JIRA for dynamic changes if
> it is ok.
> >
> >
> > > >
> > > > On Thu, Sep 3, 2015 at 1:12 AM, Atri Sharma <at...@apache.org> wrote:
> > > >
> > > > > Amol.
> > > > >
> > > > > For my understanding, when you mention launch time/code generation
> > > time,
> > > > > are you referring to generation of physical plan, please?
> > > > >
> > > > > Regards,
> > > > >
> > > > > Atri
> > > > >
> > > > > On Thu, Sep 3, 2015 at 12:48 PM, Amol Kekre <am...@datatorrent.com>
> > > wrote:
> > > > >
> > > > > > Atri,
> > > > > > For a lot of operations module should be treated as a black box.
> It
> > > is
> > > > > just
> > > > > > another reusable IP. The flattening should happen at launch time.
> > > > > >
> > > > > > If we think of Apex as a compiler, then all the compile time
> checks
> > > > > (ports
> > > > > > connectivity, matching types/schema, properties, attributes, ...)
> are
> > > as
> > > > > > applicable to modules as to operators. At launch time (aka code
> > > > > generation
> > > > > > time) module gets flattened. Webservice should still enable
> access
> > > via
> > > > > > module scope on a running app.
> > > > > >
> > > > > > Thks,
> > > > > > Amol
> > > > > >
> > > >
> > >
>
Re: Modules support in Apex
Posted by Atri Sharma <at...@gmail.com>.
> On Thu, Sep 3, 2015 at 11:55 AM, Atri Sharma <at...@gmail.com> wrote:
>
> > Thanks for the detailed explanation!
> >
> > I have few small questions. Please find them inline.
> >
> > On 3 Sep 2015 21:03, "Amol Kekre" <am...@datatorrent.com> wrote:
> >
> > > Design time -> design, try, iterate => Re-use of IP saves a lot of
time
> > > here. Operators help as leaf level IP, Modules help as higher level IP
> > that
> > > can be made by combining previously tested leaf level IP. Additionally
> > they
> > > should be (a) correct by construction due to leveraging other
operators
> > > (b) allow properties, attributes to make re-use IP log more powerful.
> > > Modules do not prohibit Apex engine from optimizing DAG during launch
(or
> > > even run) time, all the while enabling re-use at higher level. In fact
> > they
> > > may aid optimization by giving specific hints at a sub-DAG level.
Same as
> > > functions, templates, etc. in C++. This phase is mainly human time.
This
> > > results in a "logical plan" of App DAG
> > >
> >
> > So essentially a logical plan is the one manually defined by the users
> > using operators and modules, correct?
> >
>
> Yes.
Thanks!
>
> >
> > (I understand logical plan is generated from JSON files, just want to
> > clarify if the JSON files are the one defined by user to define the
DAG).
> >
> > > Launch time -> Usually a one time cost. This is always completely
> > > automated in any compiler (aka compile time). So stuffing more here
moves
> > > us from human time to computer time. Module moves a lot more work from
> > > human time to computer time as compared to leaf level operator. This
> > > results in physical plan of App DAG
> >
> > If I understand correctly, this is where we expand Modules I. E. call
> > Module's populateDAG method, correct?
> >
> >
> Yes
Thanks!
>
>
> > > Run time -> Actual app run time. At this stage the flattening has
> > already
> > > happened. Apex apps are to run for-ever or for a long time. These are
not
> > 1
> > > min apps. So an extra 10 seconds during launch time in a big data
project
> > > is amortized away over a days, months, or at least a few hours.
> >
> > Got that, thanks!
> >
> > > Operators do that at leaf level, modules enable true distributed
> > execution.
> >
> > Is the distributed nature due to reuse?
> >
> >
> Distributed nature is because Apex engine can now take this dag (called
> physical plan), get resources from RM, and create an execution plan (where
> each operator runs). Apex engine then distributes it. By flattening the
> module in logical plan to physical plan phase, we leverage Apex engine's
> in-built distributive execution. The logical plan -> physical plan ->
> execution plan is very common across a lot of distributed engines.
>
Got that, thanks.
>
> > > I am hoping to see this thesis proved by making Calcite integration
> > easier.
> >
> > I totally undrstand what Modules are meant to be and find them
fascinating.
> > I have a couple of questions around their usage please:
> >
> > 1) Is there any difference between a user using modules for reusing
static
> > DAGs vs expansion of Modules at runtime?
> > 2) After Modules are introduced, will APEX-3's objective be to have
dynamic
> > flattening of Modules at runtime?
> >
> >
> Apex-3 simply enables launch time ability, i.e static DAG. The objective
is
> to get dynamic DAG changes at module level. Currently they are at operator
> level. Their is no jira for dynamic module DAG changes akaik. But we did
> talk about it before Apex incubated.
I think my attack plans have been around dynamic DAG changes since that is
what I understood from our discussion offline earlier. I shall continue
working with Vlad in APEX 3 and shall open new JIRA for dynamic changes if
it is ok.
>
>
> > >
> > > On Thu, Sep 3, 2015 at 1:12 AM, Atri Sharma <at...@apache.org> wrote:
> > >
> > > > Amol.
> > > >
> > > > For my understanding, when you mention launch time/code generation
> > time,
> > > > are you referring to generation of physical plan, please?
> > > >
> > > > Regards,
> > > >
> > > > Atri
> > > >
> > > > On Thu, Sep 3, 2015 at 12:48 PM, Amol Kekre <am...@datatorrent.com>
> > wrote:
> > > >
> > > > > Atri,
> > > > > For a lot of operations module should be treated as a black box.
It
> > is
> > > > just
> > > > > another reusable IP. The flattening should happen at launch time.
> > > > >
> > > > > If we think of Apex as a compiler, then all the compile time
checks
> > > > (ports
> > > > > connectivity, matching types/schema, properties, attributes, ...)
are
> > as
> > > > > applicable to modules as to operators. At launch time (aka code
> > > > generation
> > > > > time) module gets flattened. Webservice should still enable access
> > via
> > > > > module scope on a running app.
> > > > >
> > > > > Thks,
> > > > > Amol
> > > > >
> > >
> >
Re: Modules support in Apex
Posted by Amol Kekre <am...@datatorrent.com>.
Atri,
I added my comments inline
Thks,
Amol
On Thu, Sep 3, 2015 at 11:55 AM, Atri Sharma <at...@gmail.com> wrote:
> Thanks for the detailed explanation!
>
> I have few small questions. Please find them inline.
>
> On 3 Sep 2015 21:03, "Amol Kekre" <am...@datatorrent.com> wrote:
>
> > Design time -> design, try, iterate => Re-use of IP saves a lot of time
> > here. Operators help as leaf level IP, Modules help as higher level IP
> that
> > can be made by combining previously tested leaf level IP. Additionally
> they
> > should be (a) correct by construction due to leveraging other operators
> > (b) allow properties, attributes to make re-use IP log more powerful.
> > Modules do not prohibit Apex engine from optimizing DAG during launch (or
> > even run) time, all the while enabling re-use at higher level. In fact
> they
> > may aid optimization by giving specific hints at a sub-DAG level. Same as
> > functions, templates, etc. in C++. This phase is mainly human time. This
> > results in a "logical plan" of App DAG
> >
>
> So essentially a logical plan is the one manually defined by the users
> using operators and modules, correct?
>
Yes.
>
> (I understand logical plan is generated from JSON files, just want to
> clarify if the JSON files are the one defined by user to define the DAG).
>
> > Launch time -> Usually a one time cost. This is always completely
> > automated in any compiler (aka compile time). So stuffing more here moves
> > us from human time to computer time. Module moves a lot more work from
> > human time to computer time as compared to leaf level operator. This
> > results in physical plan of App DAG
>
> If I understand correctly, this is where we expand Modules I. E. call
> Module's populateDAG method, correct?
>
>
Yes
> > Run time -> Actual app run time. At this stage the flattening has
> already
> > happened. Apex apps are to run for-ever or for a long time. These are not
> 1
> > min apps. So an extra 10 seconds during launch time in a big data project
> > is amortized away over a days, months, or at least a few hours.
>
> Got that, thanks!
>
> > Operators do that at leaf level, modules enable true distributed
> execution.
>
> Is the distributed nature due to reuse?
>
>
Distributed nature is because Apex engine can now take this dag (called
physical plan), get resources from RM, and create an execution plan (where
each operator runs). Apex engine then distributes it. By flattening the
module in logical plan to physical plan phase, we leverage Apex engine's
in-built distributive execution. The logical plan -> physical plan ->
execution plan is very common across a lot of distributed engines.
> > I am hoping to see this thesis proved by making Calcite integration
> easier.
>
> I totally undrstand what Modules are meant to be and find them fascinating.
> I have a couple of questions around their usage please:
>
> 1) Is there any difference between a user using modules for reusing static
> DAGs vs expansion of Modules at runtime?
> 2) After Modules are introduced, will APEX-3's objective be to have dynamic
> flattening of Modules at runtime?
>
>
Apex-3 simply enables launch time ability, i.e static DAG. The objective is
to get dynamic DAG changes at module level. Currently they are at operator
level. Their is no jira for dynamic module DAG changes akaik. But we did
talk about it before Apex incubated.
> >
> > On Thu, Sep 3, 2015 at 1:12 AM, Atri Sharma <at...@apache.org> wrote:
> >
> > > Amol.
> > >
> > > For my understanding, when you mention launch time/code generation
> time,
> > > are you referring to generation of physical plan, please?
> > >
> > > Regards,
> > >
> > > Atri
> > >
> > > On Thu, Sep 3, 2015 at 12:48 PM, Amol Kekre <am...@datatorrent.com>
> wrote:
> > >
> > > > Atri,
> > > > For a lot of operations module should be treated as a black box. It
> is
> > > just
> > > > another reusable IP. The flattening should happen at launch time.
> > > >
> > > > If we think of Apex as a compiler, then all the compile time checks
> > > (ports
> > > > connectivity, matching types/schema, properties, attributes, ...) are
> as
> > > > applicable to modules as to operators. At launch time (aka code
> > > generation
> > > > time) module gets flattened. Webservice should still enable access
> via
> > > > module scope on a running app.
> > > >
> > > > Thks,
> > > > Amol
> > > >
> >
>
Re: Modules support in Apex
Posted by Atri Sharma <at...@gmail.com>.
Thanks for the detailed explanation!
I have few small questions. Please find them inline.
On 3 Sep 2015 21:03, "Amol Kekre" <am...@datatorrent.com> wrote:
> Design time -> design, try, iterate => Re-use of IP saves a lot of time
> here. Operators help as leaf level IP, Modules help as higher level IP
that
> can be made by combining previously tested leaf level IP. Additionally
they
> should be (a) correct by construction due to leveraging other operators
> (b) allow properties, attributes to make re-use IP log more powerful.
> Modules do not prohibit Apex engine from optimizing DAG during launch (or
> even run) time, all the while enabling re-use at higher level. In fact
they
> may aid optimization by giving specific hints at a sub-DAG level. Same as
> functions, templates, etc. in C++. This phase is mainly human time. This
> results in a "logical plan" of App DAG
>
So essentially a logical plan is the one manually defined by the users
using operators and modules, correct?
(I understand logical plan is generated from JSON files, just want to
clarify if the JSON files are the one defined by user to define the DAG).
> Launch time -> Usually a one time cost. This is always completely
> automated in any compiler (aka compile time). So stuffing more here moves
> us from human time to computer time. Module moves a lot more work from
> human time to computer time as compared to leaf level operator. This
> results in physical plan of App DAG
If I understand correctly, this is where we expand Modules I. E. call
Module's populateDAG method, correct?
> Run time -> Actual app run time. At this stage the flattening has already
> happened. Apex apps are to run for-ever or for a long time. These are not
1
> min apps. So an extra 10 seconds during launch time in a big data project
> is amortized away over a days, months, or at least a few hours.
Got that, thanks!
> Operators do that at leaf level, modules enable true distributed
execution.
Is the distributed nature due to reuse?
> I am hoping to see this thesis proved by making Calcite integration
easier.
I totally undrstand what Modules are meant to be and find them fascinating.
I have a couple of questions around their usage please:
1) Is there any difference between a user using modules for reusing static
DAGs vs expansion of Modules at runtime?
2) After Modules are introduced, will APEX-3's objective be to have dynamic
flattening of Modules at runtime?
>
> On Thu, Sep 3, 2015 at 1:12 AM, Atri Sharma <at...@apache.org> wrote:
>
> > Amol.
> >
> > For my understanding, when you mention launch time/code generation time,
> > are you referring to generation of physical plan, please?
> >
> > Regards,
> >
> > Atri
> >
> > On Thu, Sep 3, 2015 at 12:48 PM, Amol Kekre <am...@datatorrent.com>
wrote:
> >
> > > Atri,
> > > For a lot of operations module should be treated as a black box. It is
> > just
> > > another reusable IP. The flattening should happen at launch time.
> > >
> > > If we think of Apex as a compiler, then all the compile time checks
> > (ports
> > > connectivity, matching types/schema, properties, attributes, ...) are
as
> > > applicable to modules as to operators. At launch time (aka code
> > generation
> > > time) module gets flattened. Webservice should still enable access via
> > > module scope on a running app.
> > >
> > > Thks,
> > > Amol
> > >
>
Re: Modules support in Apex
Posted by Amol Kekre <am...@datatorrent.com>.
Atri,
Great questions. Module flattening will mainly happen at launch time. Here
is the flow
Design time -> Launch time -> Run Time
// QA, tests, benchmarking, etc. is orthogonal to these, as each of them
will walk through the same flow
Design time -> design, try, iterate => Re-use of IP saves a lot of time
here. Operators help as leaf level IP, Modules help as higher level IP that
can be made by combining previously tested leaf level IP. Additionally they
should be (a) correct by construction due to leveraging other operators,
(b) allow properties, attributes to make re-use IP log more powerful.
Modules do not prohibit Apex engine from optimizing DAG during launch (or
even run) time, all the while enabling re-use at higher level. In fact they
may aid optimization by giving specific hints at a sub-DAG level. Same as
functions, templates, etc. in C++. This phase is mainly human time. This
results in a "logical plan" of App DAG
Launch time -> Usually a one time cost. This is always completely
automated in any compiler (aka compile time). So stuffing more here moves
us from human time to computer time. Module moves a lot more work from
human time to computer time as compared to leaf level operator. This
results in physical plan of App DAG
Run time -> Actual app run time. At this stage the flattening has already
happened. Apex apps are to run for-ever or for a long time. These are not 1
min apps. So an extra 10 seconds during launch time in a big data project
is amortized away over a days, months, or at least a few hours.
Additionally Module open up Apex engine to be used by any software that is
able to say "give me a spec, I will give back my sub-DAG". This will enable
others in the communicty to enrich/migrate app IP without Apex developers
being involved. I believe that is the ultimate gain, Module removes us
(Apex developers) from being a bottleneck in adding more IP into Apex apps.
Operators do that at leaf level, modules enable true distributed execution.
I am hoping to see this thesis proved by making Calcite integration easier.
Thks,
Amol
On Thu, Sep 3, 2015 at 1:12 AM, Atri Sharma <at...@apache.org> wrote:
> Amol.
>
> For my understanding, when you mention launch time/code generation time,
> are you referring to generation of physical plan, please?
>
> Regards,
>
> Atri
>
> On Thu, Sep 3, 2015 at 12:48 PM, Amol Kekre <am...@datatorrent.com> wrote:
>
> > Atri,
> > For a lot of operations module should be treated as a black box. It is
> just
> > another reusable IP. The flattening should happen at launch time.
> >
> > If we think of Apex as a compiler, then all the compile time checks
> (ports
> > connectivity, matching types/schema, properties, attributes, ...) are as
> > applicable to modules as to operators. At launch time (aka code
> generation
> > time) module gets flattened. Webservice should still enable access via
> > module scope on a running app.
> >
> > Thks,
> > Amol
> >
> >
> > On Thu, Sep 3, 2015 at 12:10 AM, Atri Sharma <at...@apache.org> wrote:
> >
> > > So the idea around our APEX-3 work will be that we will implement
> Module
> > > interface to build a class that adds operators at runtime? Sounds like
> a
> > > good idea, if Module is essentially a set of operators plugged in DAG.
> > >
> > > Is Module to be treated like a black box with input and output ports,
> and
> > > the internal subgraph either generated statically or dynamically?
> > >
> > > On Tue, Sep 1, 2015 at 2:47 AM, Vlad Rozov <v....@datatorrent.com>
> > > wrote:
> > >
> > > > Atri,
> > > >
> > > > As a first cut module is a predefined subgraph that can be inserted
> > into
> > > a
> > > > DAG. Generally speaking module is not required to expand to the same
> > > > subgraph. Depending on module properties or configuration, module may
> > > > expand to different subgraphs. At the same time, similar to an
> operator
> > > > module has predefined input/output ports and properties.
> > > >
> > > > Thank you,
> > > >
> > > > Vlad
> > > >
> > > >
> > > > On 8/31/15 11:46, Atri Sharma wrote:
> > > >
> > > >> No, I dont think APEX-3's functionality needs to be exceeded for
> this.
> > > >> What
> > > >> I am trying to understand here is the concept of Module. Is it a
> > family
> > > of
> > > >> operators defined by same interface implementation *or* is it a
> > defined
> > > >> subgraph that can be replaced and used as a part of a subgraph
> instead
> > > of
> > > >> building the whole connection again?
> > > >>
> > > >> On Mon, Aug 31, 2015 at 11:27 PM, Tushar Gosavi <
> > tushar@datatorrent.com
> > > >
> > > >> wrote:
> > > >>
> > > >> Yes, you are correct. The APEX-3 is for dynamic expansion of the
> DAG,
> > If
> > > >>> we
> > > >>> can
> > > >>> expand the DAG dynamically then we can use that functionality to
> > expand
> > > >>> the
> > > >>> DAG
> > > >>> with know list of operator and connections between them. In a way
> > > APEX-3
> > > >>> provides
> > > >>> bigger functionality than APEX-55.
> > > >>>
> > > >>> Let me know if you think that module functionality requires more
> > > support
> > > >>> that what will be provided by APEX-3.
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Mon, Aug 31, 2015 at 10:39 AM, Atri Sharma <at...@apache.org>
> > wrote:
> > > >>>
> > > >>> Looks interesting.
> > > >>>>
> > > >>>> Can you explain a bit on how it helps APEX-3? The objective of
> > APEX-3
> > > is
> > > >>>>
> > > >>> to
> > > >>>
> > > >>>> have dynamic expansion of sub DAGs. If I understand correctly, the
> > > >>>> static
> > > >>>> list is to be known when declaring a module.
> > > >>>>
> > > >>>> Please correct me if I am wrong.
> > > >>>>
> > > >>>> On Mon, Aug 31, 2015 at 11:05 PM, Tushar Gosavi <
> > > tushar@datatorrent.com
> > > >>>> >
> > > >>>> wrote:
> > > >>>>
> > > >>>> Hi All,
> > > >>>>>
> > > >>>>> We are working on adding support for Modules in Apex. A module
> is a
> > > >>>>>
> > > >>>> group
> > > >>>
> > > >>>> of operators that will have their own existence and will ease the
> > way
> > > >>>>>
> > > >>>> we
> > > >>>
> > > >>>> are currently defining an application.
> > > >>>>>
> > > >>>>> A module is defined by:
> > > >>>>>
> > > >>>>> - list of operators.
> > > >>>>> - list of input and output ports.
> > > >>>>> - set of properties for the module.
> > > >>>>> - set of attributes for the module.
> > > >>>>>
> > > >>>>>
> > > >>>>> Details on the proposed work is given at:
> > > >>>>>
> > > >>>>> https://malhar.atlassian.net/browse/APEX-55 . The work is also
> > > related
> > > >>>>>
> > > >>>> to
> > > >>>>
> > > >>>>> https://malhar.atlassian.net/browse/APEX-3
> > > >>>>>
> > > >>>>> Regards,
> > > >>>>> -Tushar.
> > > >>>>>
> > > >>>>>
> > > >
> > >
> >
>
Re: Modules support in Apex
Posted by Atri Sharma <at...@apache.org>.
Amol.
For my understanding, when you mention launch time/code generation time,
are you referring to generation of physical plan, please?
Regards,
Atri
On Thu, Sep 3, 2015 at 12:48 PM, Amol Kekre <am...@datatorrent.com> wrote:
> Atri,
> For a lot of operations module should be treated as a black box. It is just
> another reusable IP. The flattening should happen at launch time.
>
> If we think of Apex as a compiler, then all the compile time checks (ports
> connectivity, matching types/schema, properties, attributes, ...) are as
> applicable to modules as to operators. At launch time (aka code generation
> time) module gets flattened. Webservice should still enable access via
> module scope on a running app.
>
> Thks,
> Amol
>
>
> On Thu, Sep 3, 2015 at 12:10 AM, Atri Sharma <at...@apache.org> wrote:
>
> > So the idea around our APEX-3 work will be that we will implement Module
> > interface to build a class that adds operators at runtime? Sounds like a
> > good idea, if Module is essentially a set of operators plugged in DAG.
> >
> > Is Module to be treated like a black box with input and output ports, and
> > the internal subgraph either generated statically or dynamically?
> >
> > On Tue, Sep 1, 2015 at 2:47 AM, Vlad Rozov <v....@datatorrent.com>
> > wrote:
> >
> > > Atri,
> > >
> > > As a first cut module is a predefined subgraph that can be inserted
> into
> > a
> > > DAG. Generally speaking module is not required to expand to the same
> > > subgraph. Depending on module properties or configuration, module may
> > > expand to different subgraphs. At the same time, similar to an operator
> > > module has predefined input/output ports and properties.
> > >
> > > Thank you,
> > >
> > > Vlad
> > >
> > >
> > > On 8/31/15 11:46, Atri Sharma wrote:
> > >
> > >> No, I dont think APEX-3's functionality needs to be exceeded for this.
> > >> What
> > >> I am trying to understand here is the concept of Module. Is it a
> family
> > of
> > >> operators defined by same interface implementation *or* is it a
> defined
> > >> subgraph that can be replaced and used as a part of a subgraph instead
> > of
> > >> building the whole connection again?
> > >>
> > >> On Mon, Aug 31, 2015 at 11:27 PM, Tushar Gosavi <
> tushar@datatorrent.com
> > >
> > >> wrote:
> > >>
> > >> Yes, you are correct. The APEX-3 is for dynamic expansion of the DAG,
> If
> > >>> we
> > >>> can
> > >>> expand the DAG dynamically then we can use that functionality to
> expand
> > >>> the
> > >>> DAG
> > >>> with know list of operator and connections between them. In a way
> > APEX-3
> > >>> provides
> > >>> bigger functionality than APEX-55.
> > >>>
> > >>> Let me know if you think that module functionality requires more
> > support
> > >>> that what will be provided by APEX-3.
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Aug 31, 2015 at 10:39 AM, Atri Sharma <at...@apache.org>
> wrote:
> > >>>
> > >>> Looks interesting.
> > >>>>
> > >>>> Can you explain a bit on how it helps APEX-3? The objective of
> APEX-3
> > is
> > >>>>
> > >>> to
> > >>>
> > >>>> have dynamic expansion of sub DAGs. If I understand correctly, the
> > >>>> static
> > >>>> list is to be known when declaring a module.
> > >>>>
> > >>>> Please correct me if I am wrong.
> > >>>>
> > >>>> On Mon, Aug 31, 2015 at 11:05 PM, Tushar Gosavi <
> > tushar@datatorrent.com
> > >>>> >
> > >>>> wrote:
> > >>>>
> > >>>> Hi All,
> > >>>>>
> > >>>>> We are working on adding support for Modules in Apex. A module is a
> > >>>>>
> > >>>> group
> > >>>
> > >>>> of operators that will have their own existence and will ease the
> way
> > >>>>>
> > >>>> we
> > >>>
> > >>>> are currently defining an application.
> > >>>>>
> > >>>>> A module is defined by:
> > >>>>>
> > >>>>> - list of operators.
> > >>>>> - list of input and output ports.
> > >>>>> - set of properties for the module.
> > >>>>> - set of attributes for the module.
> > >>>>>
> > >>>>>
> > >>>>> Details on the proposed work is given at:
> > >>>>>
> > >>>>> https://malhar.atlassian.net/browse/APEX-55 . The work is also
> > related
> > >>>>>
> > >>>> to
> > >>>>
> > >>>>> https://malhar.atlassian.net/browse/APEX-3
> > >>>>>
> > >>>>> Regards,
> > >>>>> -Tushar.
> > >>>>>
> > >>>>>
> > >
> >
>
Re: Modules support in Apex
Posted by Atri Sharma <at...@apache.org>.
Hi,
Thanks for the explanation. Makes a lot more sense now.
I like the idea of Modules especially their abstraction for APEX-3. I have
few questions which I have marked inline please.
On Thu, Sep 3, 2015 at 12:48 PM, Amol Kekre <am...@datatorrent.com> wrote:
> Atri,
> For a lot of operations module should be treated as a black box. It is just
> another reusable IP. The flattening should happen at launch time.
>
Got that, thanks!
>
> If we think of Apex as a compiler, then all the compile time checks (ports
> connectivity, matching types/schema, properties, attributes, ...) are as
> applicable to modules as to operators. At launch time (aka code generation
> time) module gets flattened. Webservice should still enable access via
> module scope on a running app.
>
Ok, I was trying to think how the Module code generation will happen. Now,
we must have the ability for application developers to define modules in
sense of their operators (in terms of APEX-3).
One question though. Wont excessive usage of Modules cause a bit of a
performance drop, since Modules are essentially flattened at runtime, as
you explained, so they incur some runtime overhead?
Regards,
Atri
Re: Modules support in Apex
Posted by Amol Kekre <am...@datatorrent.com>.
Atri,
For a lot of operations module should be treated as a black box. It is just
another reusable IP. The flattening should happen at launch time.
If we think of Apex as a compiler, then all the compile time checks (ports
connectivity, matching types/schema, properties, attributes, ...) are as
applicable to modules as to operators. At launch time (aka code generation
time) module gets flattened. Webservice should still enable access via
module scope on a running app.
Thks,
Amol
On Thu, Sep 3, 2015 at 12:10 AM, Atri Sharma <at...@apache.org> wrote:
> So the idea around our APEX-3 work will be that we will implement Module
> interface to build a class that adds operators at runtime? Sounds like a
> good idea, if Module is essentially a set of operators plugged in DAG.
>
> Is Module to be treated like a black box with input and output ports, and
> the internal subgraph either generated statically or dynamically?
>
> On Tue, Sep 1, 2015 at 2:47 AM, Vlad Rozov <v....@datatorrent.com>
> wrote:
>
> > Atri,
> >
> > As a first cut module is a predefined subgraph that can be inserted into
> a
> > DAG. Generally speaking module is not required to expand to the same
> > subgraph. Depending on module properties or configuration, module may
> > expand to different subgraphs. At the same time, similar to an operator
> > module has predefined input/output ports and properties.
> >
> > Thank you,
> >
> > Vlad
> >
> >
> > On 8/31/15 11:46, Atri Sharma wrote:
> >
> >> No, I dont think APEX-3's functionality needs to be exceeded for this.
> >> What
> >> I am trying to understand here is the concept of Module. Is it a family
> of
> >> operators defined by same interface implementation *or* is it a defined
> >> subgraph that can be replaced and used as a part of a subgraph instead
> of
> >> building the whole connection again?
> >>
> >> On Mon, Aug 31, 2015 at 11:27 PM, Tushar Gosavi <tushar@datatorrent.com
> >
> >> wrote:
> >>
> >> Yes, you are correct. The APEX-3 is for dynamic expansion of the DAG, If
> >>> we
> >>> can
> >>> expand the DAG dynamically then we can use that functionality to expand
> >>> the
> >>> DAG
> >>> with know list of operator and connections between them. In a way
> APEX-3
> >>> provides
> >>> bigger functionality than APEX-55.
> >>>
> >>> Let me know if you think that module functionality requires more
> support
> >>> that what will be provided by APEX-3.
> >>>
> >>>
> >>>
> >>> On Mon, Aug 31, 2015 at 10:39 AM, Atri Sharma <at...@apache.org> wrote:
> >>>
> >>> Looks interesting.
> >>>>
> >>>> Can you explain a bit on how it helps APEX-3? The objective of APEX-3
> is
> >>>>
> >>> to
> >>>
> >>>> have dynamic expansion of sub DAGs. If I understand correctly, the
> >>>> static
> >>>> list is to be known when declaring a module.
> >>>>
> >>>> Please correct me if I am wrong.
> >>>>
> >>>> On Mon, Aug 31, 2015 at 11:05 PM, Tushar Gosavi <
> tushar@datatorrent.com
> >>>> >
> >>>> wrote:
> >>>>
> >>>> Hi All,
> >>>>>
> >>>>> We are working on adding support for Modules in Apex. A module is a
> >>>>>
> >>>> group
> >>>
> >>>> of operators that will have their own existence and will ease the way
> >>>>>
> >>>> we
> >>>
> >>>> are currently defining an application.
> >>>>>
> >>>>> A module is defined by:
> >>>>>
> >>>>> - list of operators.
> >>>>> - list of input and output ports.
> >>>>> - set of properties for the module.
> >>>>> - set of attributes for the module.
> >>>>>
> >>>>>
> >>>>> Details on the proposed work is given at:
> >>>>>
> >>>>> https://malhar.atlassian.net/browse/APEX-55 . The work is also
> related
> >>>>>
> >>>> to
> >>>>
> >>>>> https://malhar.atlassian.net/browse/APEX-3
> >>>>>
> >>>>> Regards,
> >>>>> -Tushar.
> >>>>>
> >>>>>
> >
>
Re: Modules support in Apex
Posted by Atri Sharma <at...@apache.org>.
So the idea around our APEX-3 work will be that we will implement Module
interface to build a class that adds operators at runtime? Sounds like a
good idea, if Module is essentially a set of operators plugged in DAG.
Is Module to be treated like a black box with input and output ports, and
the internal subgraph either generated statically or dynamically?
On Tue, Sep 1, 2015 at 2:47 AM, Vlad Rozov <v....@datatorrent.com> wrote:
> Atri,
>
> As a first cut module is a predefined subgraph that can be inserted into a
> DAG. Generally speaking module is not required to expand to the same
> subgraph. Depending on module properties or configuration, module may
> expand to different subgraphs. At the same time, similar to an operator
> module has predefined input/output ports and properties.
>
> Thank you,
>
> Vlad
>
>
> On 8/31/15 11:46, Atri Sharma wrote:
>
>> No, I dont think APEX-3's functionality needs to be exceeded for this.
>> What
>> I am trying to understand here is the concept of Module. Is it a family of
>> operators defined by same interface implementation *or* is it a defined
>> subgraph that can be replaced and used as a part of a subgraph instead of
>> building the whole connection again?
>>
>> On Mon, Aug 31, 2015 at 11:27 PM, Tushar Gosavi <tu...@datatorrent.com>
>> wrote:
>>
>> Yes, you are correct. The APEX-3 is for dynamic expansion of the DAG, If
>>> we
>>> can
>>> expand the DAG dynamically then we can use that functionality to expand
>>> the
>>> DAG
>>> with know list of operator and connections between them. In a way APEX-3
>>> provides
>>> bigger functionality than APEX-55.
>>>
>>> Let me know if you think that module functionality requires more support
>>> that what will be provided by APEX-3.
>>>
>>>
>>>
>>> On Mon, Aug 31, 2015 at 10:39 AM, Atri Sharma <at...@apache.org> wrote:
>>>
>>> Looks interesting.
>>>>
>>>> Can you explain a bit on how it helps APEX-3? The objective of APEX-3 is
>>>>
>>> to
>>>
>>>> have dynamic expansion of sub DAGs. If I understand correctly, the
>>>> static
>>>> list is to be known when declaring a module.
>>>>
>>>> Please correct me if I am wrong.
>>>>
>>>> On Mon, Aug 31, 2015 at 11:05 PM, Tushar Gosavi <tushar@datatorrent.com
>>>> >
>>>> wrote:
>>>>
>>>> Hi All,
>>>>>
>>>>> We are working on adding support for Modules in Apex. A module is a
>>>>>
>>>> group
>>>
>>>> of operators that will have their own existence and will ease the way
>>>>>
>>>> we
>>>
>>>> are currently defining an application.
>>>>>
>>>>> A module is defined by:
>>>>>
>>>>> - list of operators.
>>>>> - list of input and output ports.
>>>>> - set of properties for the module.
>>>>> - set of attributes for the module.
>>>>>
>>>>>
>>>>> Details on the proposed work is given at:
>>>>>
>>>>> https://malhar.atlassian.net/browse/APEX-55 . The work is also related
>>>>>
>>>> to
>>>>
>>>>> https://malhar.atlassian.net/browse/APEX-3
>>>>>
>>>>> Regards,
>>>>> -Tushar.
>>>>>
>>>>>
>
Re: Modules support in Apex
Posted by Vlad Rozov <v....@datatorrent.com>.
Hi Thomas,
I'll update JIRA after I rebase my work into the apache apex fork.
Thank you,
Vlad
On 9/2/15 14:53, Thomas Weise wrote:
> Vlad,
>
> Can you send a pointer to the fork you are working on (or mention it on the
> JIRA)?
>
> Thanks
>
> On Mon, Aug 31, 2015 at 2:17 PM, Vlad Rozov <v....@datatorrent.com> wrote:
>
>> Atri,
>>
>> As a first cut module is a predefined subgraph that can be inserted into a
>> DAG. Generally speaking module is not required to expand to the same
>> subgraph. Depending on module properties or configuration, module may
>> expand to different subgraphs. At the same time, similar to an operator
>> module has predefined input/output ports and properties.
>>
>> Thank you,
>>
>> Vlad
>>
>>
>> On 8/31/15 11:46, Atri Sharma wrote:
>>
>>> No, I dont think APEX-3's functionality needs to be exceeded for this.
>>> What
>>> I am trying to understand here is the concept of Module. Is it a family of
>>> operators defined by same interface implementation *or* is it a defined
>>> subgraph that can be replaced and used as a part of a subgraph instead of
>>> building the whole connection again?
>>>
>>> On Mon, Aug 31, 2015 at 11:27 PM, Tushar Gosavi <tu...@datatorrent.com>
>>> wrote:
>>>
>>> Yes, you are correct. The APEX-3 is for dynamic expansion of the DAG, If
>>>> we
>>>> can
>>>> expand the DAG dynamically then we can use that functionality to expand
>>>> the
>>>> DAG
>>>> with know list of operator and connections between them. In a way APEX-3
>>>> provides
>>>> bigger functionality than APEX-55.
>>>>
>>>> Let me know if you think that module functionality requires more support
>>>> that what will be provided by APEX-3.
>>>>
>>>>
>>>>
>>>> On Mon, Aug 31, 2015 at 10:39 AM, Atri Sharma <at...@apache.org> wrote:
>>>>
>>>> Looks interesting.
>>>>> Can you explain a bit on how it helps APEX-3? The objective of APEX-3 is
>>>>>
>>>> to
>>>>
>>>>> have dynamic expansion of sub DAGs. If I understand correctly, the
>>>>> static
>>>>> list is to be known when declaring a module.
>>>>>
>>>>> Please correct me if I am wrong.
>>>>>
>>>>> On Mon, Aug 31, 2015 at 11:05 PM, Tushar Gosavi <tushar@datatorrent.com
>>>>> wrote:
>>>>>
>>>>> Hi All,
>>>>>> We are working on adding support for Modules in Apex. A module is a
>>>>>>
>>>>> group
>>>>> of operators that will have their own existence and will ease the way
>>>>> we
>>>>> are currently defining an application.
>>>>>> A module is defined by:
>>>>>>
>>>>>> - list of operators.
>>>>>> - list of input and output ports.
>>>>>> - set of properties for the module.
>>>>>> - set of attributes for the module.
>>>>>>
>>>>>>
>>>>>> Details on the proposed work is given at:
>>>>>>
>>>>>> https://malhar.atlassian.net/browse/APEX-55 . The work is also related
>>>>>>
>>>>> to
>>>>>
>>>>>> https://malhar.atlassian.net/browse/APEX-3
>>>>>>
>>>>>> Regards,
>>>>>> -Tushar.
>>>>>>
>>>>>>
Re: Modules support in Apex
Posted by Thomas Weise <th...@datatorrent.com>.
Vlad,
Can you send a pointer to the fork you are working on (or mention it on the
JIRA)?
Thanks
On Mon, Aug 31, 2015 at 2:17 PM, Vlad Rozov <v....@datatorrent.com> wrote:
> Atri,
>
> As a first cut module is a predefined subgraph that can be inserted into a
> DAG. Generally speaking module is not required to expand to the same
> subgraph. Depending on module properties or configuration, module may
> expand to different subgraphs. At the same time, similar to an operator
> module has predefined input/output ports and properties.
>
> Thank you,
>
> Vlad
>
>
> On 8/31/15 11:46, Atri Sharma wrote:
>
>> No, I dont think APEX-3's functionality needs to be exceeded for this.
>> What
>> I am trying to understand here is the concept of Module. Is it a family of
>> operators defined by same interface implementation *or* is it a defined
>> subgraph that can be replaced and used as a part of a subgraph instead of
>> building the whole connection again?
>>
>> On Mon, Aug 31, 2015 at 11:27 PM, Tushar Gosavi <tu...@datatorrent.com>
>> wrote:
>>
>> Yes, you are correct. The APEX-3 is for dynamic expansion of the DAG, If
>>> we
>>> can
>>> expand the DAG dynamically then we can use that functionality to expand
>>> the
>>> DAG
>>> with know list of operator and connections between them. In a way APEX-3
>>> provides
>>> bigger functionality than APEX-55.
>>>
>>> Let me know if you think that module functionality requires more support
>>> that what will be provided by APEX-3.
>>>
>>>
>>>
>>> On Mon, Aug 31, 2015 at 10:39 AM, Atri Sharma <at...@apache.org> wrote:
>>>
>>> Looks interesting.
>>>>
>>>> Can you explain a bit on how it helps APEX-3? The objective of APEX-3 is
>>>>
>>> to
>>>
>>>> have dynamic expansion of sub DAGs. If I understand correctly, the
>>>> static
>>>> list is to be known when declaring a module.
>>>>
>>>> Please correct me if I am wrong.
>>>>
>>>> On Mon, Aug 31, 2015 at 11:05 PM, Tushar Gosavi <tushar@datatorrent.com
>>>> >
>>>> wrote:
>>>>
>>>> Hi All,
>>>>>
>>>>> We are working on adding support for Modules in Apex. A module is a
>>>>>
>>>> group
>>>
>>>> of operators that will have their own existence and will ease the way
>>>>>
>>>> we
>>>
>>>> are currently defining an application.
>>>>>
>>>>> A module is defined by:
>>>>>
>>>>> - list of operators.
>>>>> - list of input and output ports.
>>>>> - set of properties for the module.
>>>>> - set of attributes for the module.
>>>>>
>>>>>
>>>>> Details on the proposed work is given at:
>>>>>
>>>>> https://malhar.atlassian.net/browse/APEX-55 . The work is also related
>>>>>
>>>> to
>>>>
>>>>> https://malhar.atlassian.net/browse/APEX-3
>>>>>
>>>>> Regards,
>>>>> -Tushar.
>>>>>
>>>>>
>
Re: Modules support in Apex
Posted by Vlad Rozov <v....@datatorrent.com>.
Atri,
As a first cut module is a predefined subgraph that can be inserted into
a DAG. Generally speaking module is not required to expand to the same
subgraph. Depending on module properties or configuration, module may
expand to different subgraphs. At the same time, similar to an operator
module has predefined input/output ports and properties.
Thank you,
Vlad
On 8/31/15 11:46, Atri Sharma wrote:
> No, I dont think APEX-3's functionality needs to be exceeded for this. What
> I am trying to understand here is the concept of Module. Is it a family of
> operators defined by same interface implementation *or* is it a defined
> subgraph that can be replaced and used as a part of a subgraph instead of
> building the whole connection again?
>
> On Mon, Aug 31, 2015 at 11:27 PM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
>> Yes, you are correct. The APEX-3 is for dynamic expansion of the DAG, If we
>> can
>> expand the DAG dynamically then we can use that functionality to expand the
>> DAG
>> with know list of operator and connections between them. In a way APEX-3
>> provides
>> bigger functionality than APEX-55.
>>
>> Let me know if you think that module functionality requires more support
>> that what will be provided by APEX-3.
>>
>>
>>
>> On Mon, Aug 31, 2015 at 10:39 AM, Atri Sharma <at...@apache.org> wrote:
>>
>>> Looks interesting.
>>>
>>> Can you explain a bit on how it helps APEX-3? The objective of APEX-3 is
>> to
>>> have dynamic expansion of sub DAGs. If I understand correctly, the static
>>> list is to be known when declaring a module.
>>>
>>> Please correct me if I am wrong.
>>>
>>> On Mon, Aug 31, 2015 at 11:05 PM, Tushar Gosavi <tu...@datatorrent.com>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> We are working on adding support for Modules in Apex. A module is a
>> group
>>>> of operators that will have their own existence and will ease the way
>> we
>>>> are currently defining an application.
>>>>
>>>> A module is defined by:
>>>>
>>>> - list of operators.
>>>> - list of input and output ports.
>>>> - set of properties for the module.
>>>> - set of attributes for the module.
>>>>
>>>>
>>>> Details on the proposed work is given at:
>>>>
>>>> https://malhar.atlassian.net/browse/APEX-55 . The work is also related
>>> to
>>>> https://malhar.atlassian.net/browse/APEX-3
>>>>
>>>> Regards,
>>>> -Tushar.
>>>>
Re: Modules support in Apex
Posted by Atri Sharma <at...@apache.org>.
No, I dont think APEX-3's functionality needs to be exceeded for this. What
I am trying to understand here is the concept of Module. Is it a family of
operators defined by same interface implementation *or* is it a defined
subgraph that can be replaced and used as a part of a subgraph instead of
building the whole connection again?
On Mon, Aug 31, 2015 at 11:27 PM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
> Yes, you are correct. The APEX-3 is for dynamic expansion of the DAG, If we
> can
> expand the DAG dynamically then we can use that functionality to expand the
> DAG
> with know list of operator and connections between them. In a way APEX-3
> provides
> bigger functionality than APEX-55.
>
> Let me know if you think that module functionality requires more support
> that what will be provided by APEX-3.
>
>
>
> On Mon, Aug 31, 2015 at 10:39 AM, Atri Sharma <at...@apache.org> wrote:
>
> > Looks interesting.
> >
> > Can you explain a bit on how it helps APEX-3? The objective of APEX-3 is
> to
> > have dynamic expansion of sub DAGs. If I understand correctly, the static
> > list is to be known when declaring a module.
> >
> > Please correct me if I am wrong.
> >
> > On Mon, Aug 31, 2015 at 11:05 PM, Tushar Gosavi <tu...@datatorrent.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > We are working on adding support for Modules in Apex. A module is a
> group
> > > of operators that will have their own existence and will ease the way
> we
> > > are currently defining an application.
> > >
> > > A module is defined by:
> > >
> > > - list of operators.
> > > - list of input and output ports.
> > > - set of properties for the module.
> > > - set of attributes for the module.
> > >
> > >
> > > Details on the proposed work is given at:
> > >
> > > https://malhar.atlassian.net/browse/APEX-55 . The work is also related
> > to
> > > https://malhar.atlassian.net/browse/APEX-3
> > >
> > > Regards,
> > > -Tushar.
> > >
> >
>
Re: Modules support in Apex
Posted by Tushar Gosavi <tu...@datatorrent.com>.
Yes, you are correct. The APEX-3 is for dynamic expansion of the DAG, If we
can
expand the DAG dynamically then we can use that functionality to expand the
DAG
with know list of operator and connections between them. In a way APEX-3
provides
bigger functionality than APEX-55.
Let me know if you think that module functionality requires more support
that what will be provided by APEX-3.
On Mon, Aug 31, 2015 at 10:39 AM, Atri Sharma <at...@apache.org> wrote:
> Looks interesting.
>
> Can you explain a bit on how it helps APEX-3? The objective of APEX-3 is to
> have dynamic expansion of sub DAGs. If I understand correctly, the static
> list is to be known when declaring a module.
>
> Please correct me if I am wrong.
>
> On Mon, Aug 31, 2015 at 11:05 PM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
> > Hi All,
> >
> > We are working on adding support for Modules in Apex. A module is a group
> > of operators that will have their own existence and will ease the way we
> > are currently defining an application.
> >
> > A module is defined by:
> >
> > - list of operators.
> > - list of input and output ports.
> > - set of properties for the module.
> > - set of attributes for the module.
> >
> >
> > Details on the proposed work is given at:
> >
> > https://malhar.atlassian.net/browse/APEX-55 . The work is also related
> to
> > https://malhar.atlassian.net/browse/APEX-3
> >
> > Regards,
> > -Tushar.
> >
>
Re: Modules support in Apex
Posted by Atri Sharma <at...@apache.org>.
Looks interesting.
Can you explain a bit on how it helps APEX-3? The objective of APEX-3 is to
have dynamic expansion of sub DAGs. If I understand correctly, the static
list is to be known when declaring a module.
Please correct me if I am wrong.
On Mon, Aug 31, 2015 at 11:05 PM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
> Hi All,
>
> We are working on adding support for Modules in Apex. A module is a group
> of operators that will have their own existence and will ease the way we
> are currently defining an application.
>
> A module is defined by:
>
> - list of operators.
> - list of input and output ports.
> - set of properties for the module.
> - set of attributes for the module.
>
>
> Details on the proposed work is given at:
>
> https://malhar.atlassian.net/browse/APEX-55 . The work is also related to
> https://malhar.atlassian.net/browse/APEX-3
>
> Regards,
> -Tushar.
>