You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Tushar Gosavi <tu...@datatorrent.com> on 2015/10/12 13:40:06 UTC

[APEX-105] Support for specifying module properties on modules.

Hi All,

As part of module support we will allow specifying module properties
through
external configuration files. The format will be similar to the operator,
instead of operator keyword we will use module keyword for properties
on module.

dt.application.<appname>.module.<modulename>.prop.<propertyname>=<value>
dt.application.<appname>.module.<modulename>.<propertyname>=<value>
dt.module.<modulename>.prop.<propertyname>=<value>
dt.module.<modulename>.<propertyname>=<value>


Setting attribute on the module.
dt.application.<appname>.module.<modulename>.attr.<moduleattr>=<value>
dt.module.<modulename>.attr.<moduleattr>=<value>

There are no module attributes defined now, but we will add them in future
like
whether to group operators during parallel partitioning.

Setting attribute on a module port.
dt.module.<modulename>.port.<portname>.attr.<attributename>

The port attributes will be transferred to the internal operator port where
it is
mapped to.

Please let us know, If we are missing something, or some syntax need
changes.

Regards,
- Tushar.

Re: [APEX-105] Support for specifying module properties on modules.

Posted by David Yan <da...@datatorrent.com>.
I think the way of specifying the property without the "prop" keyword has
been deprecated.  Maybe we should not include that support in new code.

David

On Mon, Oct 12, 2015 at 4:40 AM, Tushar Gosavi <tu...@datatorrent.com>
wrote:

> Hi All,
>
> As part of module support we will allow specifying module properties
> through
> external configuration files. The format will be similar to the operator,
> instead of operator keyword we will use module keyword for properties
> on module.
>
> dt.application.<appname>.module.<modulename>.prop.<propertyname>=<value>
> dt.application.<appname>.module.<modulename>.<propertyname>=<value>
> dt.module.<modulename>.prop.<propertyname>=<value>
> dt.module.<modulename>.<propertyname>=<value>
>
>
> Setting attribute on the module.
> dt.application.<appname>.module.<modulename>.attr.<moduleattr>=<value>
> dt.module.<modulename>.attr.<moduleattr>=<value>
>
> There are no module attributes defined now, but we will add them in future
> like
> whether to group operators during parallel partitioning.
>
> Setting attribute on a module port.
> dt.module.<modulename>.port.<portname>.attr.<attributename>
>
> The port attributes will be transferred to the internal operator port where
> it is
> mapped to.
>
> Please let us know, If we are missing something, or some syntax need
> changes.
>
> Regards,
> - Tushar.
>

Re: [APEX-105] Support for specifying module properties on modules.

Posted by Tushar Gosavi <tu...@datatorrent.com>.
A module is superset of operator before expansion in logical plan, except
operator
attribute are not applicable to modules. As Tim suggested we will have to
add checks
later to validate these attributes settings, and not allow top level
operator and
module to share same name.

The same question comes when we want to add module to application using
property
file API and JSON file API.

dt.operator.<operatorname>.classname=<fqcn of operator>

One approach we can take is, if class is of type module, then use addModule
api
to add module to the dag, else use addOperator api. The initialization of
object and applying property is generic can be reused by module.

Similarly we have Json API where all operators are specified under
operators array

{
  "operators": [
    {.. operators .. }
  ],
  "streams": [
    {.. streams .. }
  ]
}

We could add module in the operators array, and based on the object type we
can
consider it as module or operator. The mechanism of initializing object and
applying
properties remains same.

- Tushar.

Re: [APEX-105] Support for specifying module properties on modules.

Posted by Thomas Weise <th...@datatorrent.com>.
+1 If module is intended to be a superset of operator, then it should be
usable like an operator for the configuration aspect also.

On Mon, Oct 12, 2015 at 12:50 PM, Vlad Rozov <v....@datatorrent.com>
wrote:

> I think that we should try make modules interchangeable with operators as
> much as possible, so that applications (the same applies to modules)
> designer can replace a module with a functional equivalent operator or
> replace an operator with a functional equivalent module. In certain context
> modules should behave like operators and be indistinguishable from
> operators. Taking this into account, I think that using a module and an
> operator with the same name within DAG namespace should not be allowed. It
> still should be possible to have an operator in the DAG with the same name
> as another operator added to a module belonging to the DAG as module's DAG
> has its own namespace.
>
> Thank you,
>
> Vlad
>
>
> On 10/12/15 12:30, Timothy Farkas wrote:
>
>> I think using the module keyword in configurations helps to reduce
>> confusing corner cases. There are four cases/things I don't like about
>> using the operator keyword for both operators and modules:
>>
>> 1. A module with the same name as one of its internal operators
>> 2. A module with the same name as another operator in the dag.
>> 3. When there are module specific attributes, things could get
>> unnecessarily messy in the configuration handling code
>> 4. Confusing the user by giving them the impression that modules and
>> operators are the same thing when they're not.
>>
>> Thanks,
>> Tim
>>
>> On Mon, Oct 12, 2015 at 10:08 AM, Tushar Gosavi <tu...@datatorrent.com>
>> wrote:
>>
>> I was thinking from more of the view that module and operator can share a
>>> same name in the DAG. In this case it is necessary to distinguish between
>>> operator and module. If we don't want this behavior, and if its ok to use
>>> operator keyword to set module property then we can treat them as
>>> operators
>>> while setting properties, In fact, I think by doing this we could reuse
>>> property setting code for operator.
>>>
>>> - Tushar.
>>>
>>>
>>> On Mon, Oct 12, 2015 at 9:52 PM, Vlad Rozov <v....@datatorrent.com>
>>> wrote:
>>>
>>> Is it necessary to distinguish modules from operators when setting
>>>>
>>> modules
>>>
>>>> properties? Unless it is necessary to set module internal operator
>>>> properties it may be better to treat modules the same way as operators.
>>>>
>>>> Thank you,
>>>>
>>>> Vlad
>>>>
>>>>
>>>> On 10/12/15 04:40, Tushar Gosavi wrote:
>>>>
>>>> Hi All,
>>>>>
>>>>> As part of module support we will allow specifying module properties
>>>>> through
>>>>> external configuration files. The format will be similar to the
>>>>>
>>>> operator,
>>>
>>>> instead of operator keyword we will use module keyword for properties
>>>>> on module.
>>>>>
>>>>>
>>>>> dt.application.<appname>.module.<modulename>.prop.<propertyname>=<value>
>>>>> dt.application.<appname>.module.<modulename>.<propertyname>=<value>
>>>>> dt.module.<modulename>.prop.<propertyname>=<value>
>>>>> dt.module.<modulename>.<propertyname>=<value>
>>>>>
>>>>>
>>>>> Setting attribute on the module.
>>>>> dt.application.<appname>.module.<modulename>.attr.<moduleattr>=<value>
>>>>> dt.module.<modulename>.attr.<moduleattr>=<value>
>>>>>
>>>>> There are no module attributes defined now, but we will add them in
>>>>>
>>>> future
>>>
>>>> like
>>>>> whether to group operators during parallel partitioning.
>>>>>
>>>>> Setting attribute on a module port.
>>>>> dt.module.<modulename>.port.<portname>.attr.<attributename>
>>>>>
>>>>> The port attributes will be transferred to the internal operator port
>>>>> where
>>>>> it is
>>>>> mapped to.
>>>>>
>>>>> Please let us know, If we are missing something, or some syntax need
>>>>> changes.
>>>>>
>>>>> Regards,
>>>>> - Tushar.
>>>>>
>>>>>
>>>>>
>

Re: [APEX-105] Support for specifying module properties on modules.

Posted by Vlad Rozov <v....@datatorrent.com>.
I think that we should try make modules interchangeable with operators 
as much as possible, so that applications (the same applies to modules) 
designer can replace a module with a functional equivalent operator or 
replace an operator with a functional equivalent module. In certain 
context modules should behave like operators and be indistinguishable 
from operators. Taking this into account, I think that using a module 
and an operator with the same name within DAG namespace should not be 
allowed. It still should be possible to have an operator in the DAG with 
the same name as another operator added to a module belonging to the DAG 
as module's DAG has its own namespace.

Thank you,

Vlad

On 10/12/15 12:30, Timothy Farkas wrote:
> I think using the module keyword in configurations helps to reduce
> confusing corner cases. There are four cases/things I don't like about
> using the operator keyword for both operators and modules:
>
> 1. A module with the same name as one of its internal operators
> 2. A module with the same name as another operator in the dag.
> 3. When there are module specific attributes, things could get
> unnecessarily messy in the configuration handling code
> 4. Confusing the user by giving them the impression that modules and
> operators are the same thing when they're not.
>
> Thanks,
> Tim
>
> On Mon, Oct 12, 2015 at 10:08 AM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
>> I was thinking from more of the view that module and operator can share a
>> same name in the DAG. In this case it is necessary to distinguish between
>> operator and module. If we don't want this behavior, and if its ok to use
>> operator keyword to set module property then we can treat them as operators
>> while setting properties, In fact, I think by doing this we could reuse
>> property setting code for operator.
>>
>> - Tushar.
>>
>>
>> On Mon, Oct 12, 2015 at 9:52 PM, Vlad Rozov <v....@datatorrent.com>
>> wrote:
>>
>>> Is it necessary to distinguish modules from operators when setting
>> modules
>>> properties? Unless it is necessary to set module internal operator
>>> properties it may be better to treat modules the same way as operators.
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>>
>>> On 10/12/15 04:40, Tushar Gosavi wrote:
>>>
>>>> Hi All,
>>>>
>>>> As part of module support we will allow specifying module properties
>>>> through
>>>> external configuration files. The format will be similar to the
>> operator,
>>>> instead of operator keyword we will use module keyword for properties
>>>> on module.
>>>>
>>>> dt.application.<appname>.module.<modulename>.prop.<propertyname>=<value>
>>>> dt.application.<appname>.module.<modulename>.<propertyname>=<value>
>>>> dt.module.<modulename>.prop.<propertyname>=<value>
>>>> dt.module.<modulename>.<propertyname>=<value>
>>>>
>>>>
>>>> Setting attribute on the module.
>>>> dt.application.<appname>.module.<modulename>.attr.<moduleattr>=<value>
>>>> dt.module.<modulename>.attr.<moduleattr>=<value>
>>>>
>>>> There are no module attributes defined now, but we will add them in
>> future
>>>> like
>>>> whether to group operators during parallel partitioning.
>>>>
>>>> Setting attribute on a module port.
>>>> dt.module.<modulename>.port.<portname>.attr.<attributename>
>>>>
>>>> The port attributes will be transferred to the internal operator port
>>>> where
>>>> it is
>>>> mapped to.
>>>>
>>>> Please let us know, If we are missing something, or some syntax need
>>>> changes.
>>>>
>>>> Regards,
>>>> - Tushar.
>>>>
>>>>


Re: [APEX-105] Support for specifying module properties on modules.

Posted by Timothy Farkas <ti...@datatorrent.com>.
I think using the module keyword in configurations helps to reduce
confusing corner cases. There are four cases/things I don't like about
using the operator keyword for both operators and modules:

1. A module with the same name as one of its internal operators
2. A module with the same name as another operator in the dag.
3. When there are module specific attributes, things could get
unnecessarily messy in the configuration handling code
4. Confusing the user by giving them the impression that modules and
operators are the same thing when they're not.

Thanks,
Tim

On Mon, Oct 12, 2015 at 10:08 AM, Tushar Gosavi <tu...@datatorrent.com>
wrote:

> I was thinking from more of the view that module and operator can share a
> same name in the DAG. In this case it is necessary to distinguish between
> operator and module. If we don't want this behavior, and if its ok to use
> operator keyword to set module property then we can treat them as operators
> while setting properties, In fact, I think by doing this we could reuse
> property setting code for operator.
>
> - Tushar.
>
>
> On Mon, Oct 12, 2015 at 9:52 PM, Vlad Rozov <v....@datatorrent.com>
> wrote:
>
> > Is it necessary to distinguish modules from operators when setting
> modules
> > properties? Unless it is necessary to set module internal operator
> > properties it may be better to treat modules the same way as operators.
> >
> > Thank you,
> >
> > Vlad
> >
> >
> > On 10/12/15 04:40, Tushar Gosavi wrote:
> >
> >> Hi All,
> >>
> >> As part of module support we will allow specifying module properties
> >> through
> >> external configuration files. The format will be similar to the
> operator,
> >> instead of operator keyword we will use module keyword for properties
> >> on module.
> >>
> >> dt.application.<appname>.module.<modulename>.prop.<propertyname>=<value>
> >> dt.application.<appname>.module.<modulename>.<propertyname>=<value>
> >> dt.module.<modulename>.prop.<propertyname>=<value>
> >> dt.module.<modulename>.<propertyname>=<value>
> >>
> >>
> >> Setting attribute on the module.
> >> dt.application.<appname>.module.<modulename>.attr.<moduleattr>=<value>
> >> dt.module.<modulename>.attr.<moduleattr>=<value>
> >>
> >> There are no module attributes defined now, but we will add them in
> future
> >> like
> >> whether to group operators during parallel partitioning.
> >>
> >> Setting attribute on a module port.
> >> dt.module.<modulename>.port.<portname>.attr.<attributename>
> >>
> >> The port attributes will be transferred to the internal operator port
> >> where
> >> it is
> >> mapped to.
> >>
> >> Please let us know, If we are missing something, or some syntax need
> >> changes.
> >>
> >> Regards,
> >> - Tushar.
> >>
> >>
> >
>

Re: [APEX-105] Support for specifying module properties on modules.

Posted by Tushar Gosavi <tu...@datatorrent.com>.
I was thinking from more of the view that module and operator can share a
same name in the DAG. In this case it is necessary to distinguish between
operator and module. If we don't want this behavior, and if its ok to use
operator keyword to set module property then we can treat them as operators
while setting properties, In fact, I think by doing this we could reuse
property setting code for operator.

- Tushar.


On Mon, Oct 12, 2015 at 9:52 PM, Vlad Rozov <v....@datatorrent.com> wrote:

> Is it necessary to distinguish modules from operators when setting modules
> properties? Unless it is necessary to set module internal operator
> properties it may be better to treat modules the same way as operators.
>
> Thank you,
>
> Vlad
>
>
> On 10/12/15 04:40, Tushar Gosavi wrote:
>
>> Hi All,
>>
>> As part of module support we will allow specifying module properties
>> through
>> external configuration files. The format will be similar to the operator,
>> instead of operator keyword we will use module keyword for properties
>> on module.
>>
>> dt.application.<appname>.module.<modulename>.prop.<propertyname>=<value>
>> dt.application.<appname>.module.<modulename>.<propertyname>=<value>
>> dt.module.<modulename>.prop.<propertyname>=<value>
>> dt.module.<modulename>.<propertyname>=<value>
>>
>>
>> Setting attribute on the module.
>> dt.application.<appname>.module.<modulename>.attr.<moduleattr>=<value>
>> dt.module.<modulename>.attr.<moduleattr>=<value>
>>
>> There are no module attributes defined now, but we will add them in future
>> like
>> whether to group operators during parallel partitioning.
>>
>> Setting attribute on a module port.
>> dt.module.<modulename>.port.<portname>.attr.<attributename>
>>
>> The port attributes will be transferred to the internal operator port
>> where
>> it is
>> mapped to.
>>
>> Please let us know, If we are missing something, or some syntax need
>> changes.
>>
>> Regards,
>> - Tushar.
>>
>>
>

Re: [APEX-105] Support for specifying module properties on modules.

Posted by Vlad Rozov <v....@datatorrent.com>.
Is it necessary to distinguish modules from operators when setting 
modules properties? Unless it is necessary to set module internal 
operator properties it may be better to treat modules the same way as 
operators.

Thank you,

Vlad

On 10/12/15 04:40, Tushar Gosavi wrote:
> Hi All,
>
> As part of module support we will allow specifying module properties
> through
> external configuration files. The format will be similar to the operator,
> instead of operator keyword we will use module keyword for properties
> on module.
>
> dt.application.<appname>.module.<modulename>.prop.<propertyname>=<value>
> dt.application.<appname>.module.<modulename>.<propertyname>=<value>
> dt.module.<modulename>.prop.<propertyname>=<value>
> dt.module.<modulename>.<propertyname>=<value>
>
>
> Setting attribute on the module.
> dt.application.<appname>.module.<modulename>.attr.<moduleattr>=<value>
> dt.module.<modulename>.attr.<moduleattr>=<value>
>
> There are no module attributes defined now, but we will add them in future
> like
> whether to group operators during parallel partitioning.
>
> Setting attribute on a module port.
> dt.module.<modulename>.port.<portname>.attr.<attributename>
>
> The port attributes will be transferred to the internal operator port where
> it is
> mapped to.
>
> Please let us know, If we are missing something, or some syntax need
> changes.
>
> Regards,
> - Tushar.
>