You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Tushar Gosavi <tu...@datatorrent.com> on 2015/11/30 17:08:54 UTC
Re: [APEX-107] Support for specifying module properties on modules.
Hi All,
I am going to add a feature to allow use of modules in property file and
json api.
I am planing to use same "operators" array for adding modules as well as
operators. The reasoning behind is that user don't have to worry about the
component being added to the DAG is actually an operator or a module, as
the method to specify their properties and to create instance of them is
same. And it results in less complex syntax for writing application using
json/property file.
For example the sample Application specified using Json will look like
below.
```json
{
"operators": [
{
"name": "operator1",
"class": "com.datatorrent.lib.operator.Input",
"properties": {
"property1": "value1"
}
},
{
"name": "module1",
"class": "com.datatorrent.module.Module1",
"properties": {
"property1": "value1"
}
},
],
"streams": [
{
"name": "s1",
"source": {
"operatorName": "operator1",
"portName": "output"
},
"sinks": [
{
"operatorName": "module",
"portName": "input"
}
]
}
]
}
```
In above example "module1" is of type Module. While processing the Json if
the
class of component is of type Module then we will use addModule api to the
DAG.
Let me know, what do you think about this.
Regards,
- Tushar.
On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
>
> A module is superset of operator before expansion in logical plan, except
> operator
> attribute are not applicable to modules. As Tim suggested we will have to
> add checks
> later to validate these attributes settings, and not allow top level
> operator and
> module to share same name.
>
> The same question comes when we want to add module to application using
> property
> file API and JSON file API.
>
> dt.operator.<operatorname>.classname=<fqcn of operator>
>
> One approach we can take is, if class is of type module, then use
> addModule api
> to add module to the dag, else use addOperator api. The initialization of
> object and applying property is generic can be reused by module.
>
> Similarly we have Json API where all operators are specified under
> operators array
>
> {
> "operators": [
> {.. operators .. }
> ],
> "streams": [
> {.. streams .. }
> ]
> }
>
> We could add module in the operators array, and based on the object type
> we can
> consider it as module or operator. The mechanism of initializing object
> and applying
> properties remains same.
>
> - Tushar.
>
>
>
>
>
Re: [APEX-107] Support for specifying module properties on modules.
Posted by Sandeep Deshmukh <sa...@datatorrent.com>.
+1 for modules and operators to be used interchangeably.
Regards,
Sandeep
On Tue, Dec 1, 2015 at 11:54 PM, Vlad Rozov <v....@datatorrent.com> wrote:
> We should either provide Java API that supports interchangeability between
> operators and modules or ignore how an operator/module was added to a DAG.
>
> Thank you,
>
> Vlad
>
>
> On 11/30/15 20:21, Tushar Gosavi wrote:
>
>> Good point Vlad, if the class implements both Operator and Module
>> interface. We will consider it as a module by default in json API, But
>> also
>> provide a way to specify his intention whether it should be considered as
>> Module or Operator.
>>
>> If user use a Java API then intention is clear whether the object should
>> be
>> considered as Module or Operator (addOperator v/s addModule), will it be
>> correct to change the intention of the user internally based on the type
>> of
>> component?
>>
>> - Tushar.
>>
>>
>> On Tue, Dec 1, 2015 at 8:34 AM, Vlad Rozov <v....@datatorrent.com>
>> wrote:
>>
>> +1. Modules and operators should be interchangeable. It should be possible
>>> for an operator designer to replace it with a module without requirement
>>> to
>>> recompile all applications that use the existing operator. From
>>> application
>>> designer view both operators and modules may be considered as a black box
>>> with a private implementation that may be monolithic (operator) or
>>> distributed (module).
>>>
>>> Note that a Java class may implement both Operator and Module interface
>>> and be added to a DAG using addOperator() method. At run-time (Logical
>>> Plan->Physical Plan) Apex platform should handle such classes as modules
>>> and may not depend on how a module was added to a DAG, IMO.
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>>
>>> On 11/30/15 08:41, Tushar Gosavi wrote:
>>>
>>> addOperator method in DAG interface takes object of type Operator, unless
>>>> we make Module as subclass of Operator we can not use the same API.
>>>>
>>>> - Tushar.
>>>>
>>>>
>>>> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <th...@datatorrent.com>
>>>> wrote:
>>>>
>>>> +1
>>>>
>>>>> If there are really module specific constructs, they can be added
>>>>> later.
>>>>>
>>>>> Why is the same not supported in the Java API, i.e. why not allow a
>>>>> module
>>>>> to be added like an operator and handle the distinction in the
>>>>> implementation?
>>>>>
>>>>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tushar@datatorrent.com
>>>>> >
>>>>> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>>> I am going to add a feature to allow use of modules in property file
>>>>>> and
>>>>>> json api.
>>>>>> I am planing to use same "operators" array for adding modules as well
>>>>>> as
>>>>>> operators. The reasoning behind is that user don't have to worry about
>>>>>>
>>>>>> the
>>>>>
>>>>> component being added to the DAG is actually an operator or a module,
>>>>>> as
>>>>>> the method to specify their properties and to create instance of them
>>>>>> is
>>>>>> same. And it results in less complex syntax for writing application
>>>>>> using
>>>>>> json/property file.
>>>>>>
>>>>>> For example the sample Application specified using Json will look like
>>>>>> below.
>>>>>> ```json
>>>>>> {
>>>>>> "operators": [
>>>>>> {
>>>>>> "name": "operator1",
>>>>>> "class": "com.datatorrent.lib.operator.Input",
>>>>>> "properties": {
>>>>>> "property1": "value1"
>>>>>> }
>>>>>> },
>>>>>> {
>>>>>> "name": "module1",
>>>>>> "class": "com.datatorrent.module.Module1",
>>>>>> "properties": {
>>>>>> "property1": "value1"
>>>>>> }
>>>>>> },
>>>>>> ],
>>>>>> "streams": [
>>>>>> {
>>>>>> "name": "s1",
>>>>>> "source": {
>>>>>> "operatorName": "operator1",
>>>>>> "portName": "output"
>>>>>> },
>>>>>> "sinks": [
>>>>>> {
>>>>>> "operatorName": "module",
>>>>>> "portName": "input"
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> ```
>>>>>>
>>>>>> In above example "module1" is of type Module. While processing the
>>>>>> Json
>>>>>>
>>>>>> if
>>>>>
>>>>> the
>>>>>> class of component is of type Module then we will use addModule api to
>>>>>>
>>>>>> the
>>>>>
>>>>> DAG.
>>>>>>
>>>>>> Let me know, what do you think about this.
>>>>>>
>>>>>> Regards,
>>>>>> - Tushar.
>>>>>>
>>>>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <
>>>>>> tushar@datatorrent.com>
>>>>>> wrote:
>>>>>>
>>>>>> A module is superset of operator before expansion in logical plan,
>>>>>> except
>>>>>> operator
>>>>>>
>>>>>>> attribute are not applicable to modules. As Tim suggested we will
>>>>>>> have
>>>>>>>
>>>>>>> to
>>>>>> add checks
>>>>>>
>>>>>>> later to validate these attributes settings, and not allow top level
>>>>>>> operator and
>>>>>>> module to share same name.
>>>>>>>
>>>>>>> The same question comes when we want to add module to application
>>>>>>> using
>>>>>>> property
>>>>>>> file API and JSON file API.
>>>>>>>
>>>>>>> dt.operator.<operatorname>.classname=<fqcn of operator>
>>>>>>>
>>>>>>> One approach we can take is, if class is of type module, then use
>>>>>>> addModule api
>>>>>>> to add module to the dag, else use addOperator api. The
>>>>>>> initialization
>>>>>>>
>>>>>>> of
>>>>>> object and applying property is generic can be reused by module.
>>>>>>
>>>>>>> Similarly we have Json API where all operators are specified under
>>>>>>> operators array
>>>>>>>
>>>>>>> {
>>>>>>> "operators": [
>>>>>>> {.. operators .. }
>>>>>>> ],
>>>>>>> "streams": [
>>>>>>> {.. streams .. }
>>>>>>> ]
>>>>>>> }
>>>>>>>
>>>>>>> We could add module in the operators array, and based on the object
>>>>>>>
>>>>>>> type
>>>>>> we can
>>>>>>
>>>>>>> consider it as module or operator. The mechanism of initializing
>>>>>>> object
>>>>>>> and applying
>>>>>>> properties remains same.
>>>>>>>
>>>>>>> - Tushar.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>
Re: [APEX-107] Support for specifying module properties on modules.
Posted by Tushar Gosavi <tu...@datatorrent.com>.
Thanks Vlad, I will check if we can override addOperator in DAG to accept
Module too. This way we can have similar API.
- Tushar.
On Tue, Dec 1, 2015 at 11:54 PM, Vlad Rozov <v....@datatorrent.com> wrote:
> We should either provide Java API that supports interchangeability between
> operators and modules or ignore how an operator/module was added to a DAG.
>
> Thank you,
>
> Vlad
>
>
> On 11/30/15 20:21, Tushar Gosavi wrote:
>
>> Good point Vlad, if the class implements both Operator and Module
>> interface. We will consider it as a module by default in json API, But
>> also
>> provide a way to specify his intention whether it should be considered as
>> Module or Operator.
>>
>> If user use a Java API then intention is clear whether the object should
>> be
>> considered as Module or Operator (addOperator v/s addModule), will it be
>> correct to change the intention of the user internally based on the type
>> of
>> component?
>>
>> - Tushar.
>>
>>
>> On Tue, Dec 1, 2015 at 8:34 AM, Vlad Rozov <v....@datatorrent.com>
>> wrote:
>>
>> +1. Modules and operators should be interchangeable. It should be possible
>>> for an operator designer to replace it with a module without requirement
>>> to
>>> recompile all applications that use the existing operator. From
>>> application
>>> designer view both operators and modules may be considered as a black box
>>> with a private implementation that may be monolithic (operator) or
>>> distributed (module).
>>>
>>> Note that a Java class may implement both Operator and Module interface
>>> and be added to a DAG using addOperator() method. At run-time (Logical
>>> Plan->Physical Plan) Apex platform should handle such classes as modules
>>> and may not depend on how a module was added to a DAG, IMO.
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>>
>>> On 11/30/15 08:41, Tushar Gosavi wrote:
>>>
>>> addOperator method in DAG interface takes object of type Operator, unless
>>>> we make Module as subclass of Operator we can not use the same API.
>>>>
>>>> - Tushar.
>>>>
>>>>
>>>> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <th...@datatorrent.com>
>>>> wrote:
>>>>
>>>> +1
>>>>
>>>>> If there are really module specific constructs, they can be added
>>>>> later.
>>>>>
>>>>> Why is the same not supported in the Java API, i.e. why not allow a
>>>>> module
>>>>> to be added like an operator and handle the distinction in the
>>>>> implementation?
>>>>>
>>>>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tushar@datatorrent.com
>>>>> >
>>>>> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>>> I am going to add a feature to allow use of modules in property file
>>>>>> and
>>>>>> json api.
>>>>>> I am planing to use same "operators" array for adding modules as well
>>>>>> as
>>>>>> operators. The reasoning behind is that user don't have to worry about
>>>>>>
>>>>>> the
>>>>>
>>>>> component being added to the DAG is actually an operator or a module,
>>>>>> as
>>>>>> the method to specify their properties and to create instance of them
>>>>>> is
>>>>>> same. And it results in less complex syntax for writing application
>>>>>> using
>>>>>> json/property file.
>>>>>>
>>>>>> For example the sample Application specified using Json will look like
>>>>>> below.
>>>>>> ```json
>>>>>> {
>>>>>> "operators": [
>>>>>> {
>>>>>> "name": "operator1",
>>>>>> "class": "com.datatorrent.lib.operator.Input",
>>>>>> "properties": {
>>>>>> "property1": "value1"
>>>>>> }
>>>>>> },
>>>>>> {
>>>>>> "name": "module1",
>>>>>> "class": "com.datatorrent.module.Module1",
>>>>>> "properties": {
>>>>>> "property1": "value1"
>>>>>> }
>>>>>> },
>>>>>> ],
>>>>>> "streams": [
>>>>>> {
>>>>>> "name": "s1",
>>>>>> "source": {
>>>>>> "operatorName": "operator1",
>>>>>> "portName": "output"
>>>>>> },
>>>>>> "sinks": [
>>>>>> {
>>>>>> "operatorName": "module",
>>>>>> "portName": "input"
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> ```
>>>>>>
>>>>>> In above example "module1" is of type Module. While processing the
>>>>>> Json
>>>>>>
>>>>>> if
>>>>>
>>>>> the
>>>>>> class of component is of type Module then we will use addModule api to
>>>>>>
>>>>>> the
>>>>>
>>>>> DAG.
>>>>>>
>>>>>> Let me know, what do you think about this.
>>>>>>
>>>>>> Regards,
>>>>>> - Tushar.
>>>>>>
>>>>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <
>>>>>> tushar@datatorrent.com>
>>>>>> wrote:
>>>>>>
>>>>>> A module is superset of operator before expansion in logical plan,
>>>>>> except
>>>>>> operator
>>>>>>
>>>>>>> attribute are not applicable to modules. As Tim suggested we will
>>>>>>> have
>>>>>>>
>>>>>>> to
>>>>>> add checks
>>>>>>
>>>>>>> later to validate these attributes settings, and not allow top level
>>>>>>> operator and
>>>>>>> module to share same name.
>>>>>>>
>>>>>>> The same question comes when we want to add module to application
>>>>>>> using
>>>>>>> property
>>>>>>> file API and JSON file API.
>>>>>>>
>>>>>>> dt.operator.<operatorname>.classname=<fqcn of operator>
>>>>>>>
>>>>>>> One approach we can take is, if class is of type module, then use
>>>>>>> addModule api
>>>>>>> to add module to the dag, else use addOperator api. The
>>>>>>> initialization
>>>>>>>
>>>>>>> of
>>>>>> object and applying property is generic can be reused by module.
>>>>>>
>>>>>>> Similarly we have Json API where all operators are specified under
>>>>>>> operators array
>>>>>>>
>>>>>>> {
>>>>>>> "operators": [
>>>>>>> {.. operators .. }
>>>>>>> ],
>>>>>>> "streams": [
>>>>>>> {.. streams .. }
>>>>>>> ]
>>>>>>> }
>>>>>>>
>>>>>>> We could add module in the operators array, and based on the object
>>>>>>>
>>>>>>> type
>>>>>> we can
>>>>>>
>>>>>>> consider it as module or operator. The mechanism of initializing
>>>>>>> object
>>>>>>> and applying
>>>>>>> properties remains same.
>>>>>>>
>>>>>>> - Tushar.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>
Re: [APEX-107] Support for specifying module properties on modules.
Posted by Vlad Rozov <v....@datatorrent.com>.
We should either provide Java API that supports interchangeability
between operators and modules or ignore how an operator/module was added
to a DAG.
Thank you,
Vlad
On 11/30/15 20:21, Tushar Gosavi wrote:
> Good point Vlad, if the class implements both Operator and Module
> interface. We will consider it as a module by default in json API, But also
> provide a way to specify his intention whether it should be considered as
> Module or Operator.
>
> If user use a Java API then intention is clear whether the object should be
> considered as Module or Operator (addOperator v/s addModule), will it be
> correct to change the intention of the user internally based on the type of
> component?
>
> - Tushar.
>
>
> On Tue, Dec 1, 2015 at 8:34 AM, Vlad Rozov <v....@datatorrent.com> wrote:
>
>> +1. Modules and operators should be interchangeable. It should be possible
>> for an operator designer to replace it with a module without requirement to
>> recompile all applications that use the existing operator. From application
>> designer view both operators and modules may be considered as a black box
>> with a private implementation that may be monolithic (operator) or
>> distributed (module).
>>
>> Note that a Java class may implement both Operator and Module interface
>> and be added to a DAG using addOperator() method. At run-time (Logical
>> Plan->Physical Plan) Apex platform should handle such classes as modules
>> and may not depend on how a module was added to a DAG, IMO.
>>
>> Thank you,
>>
>> Vlad
>>
>>
>> On 11/30/15 08:41, Tushar Gosavi wrote:
>>
>>> addOperator method in DAG interface takes object of type Operator, unless
>>> we make Module as subclass of Operator we can not use the same API.
>>>
>>> - Tushar.
>>>
>>>
>>> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <th...@datatorrent.com>
>>> wrote:
>>>
>>> +1
>>>> If there are really module specific constructs, they can be added later.
>>>>
>>>> Why is the same not supported in the Java API, i.e. why not allow a
>>>> module
>>>> to be added like an operator and handle the distinction in the
>>>> implementation?
>>>>
>>>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tu...@datatorrent.com>
>>>> wrote:
>>>>
>>>> Hi All,
>>>>> I am going to add a feature to allow use of modules in property file and
>>>>> json api.
>>>>> I am planing to use same "operators" array for adding modules as well as
>>>>> operators. The reasoning behind is that user don't have to worry about
>>>>>
>>>> the
>>>>
>>>>> component being added to the DAG is actually an operator or a module, as
>>>>> the method to specify their properties and to create instance of them is
>>>>> same. And it results in less complex syntax for writing application
>>>>> using
>>>>> json/property file.
>>>>>
>>>>> For example the sample Application specified using Json will look like
>>>>> below.
>>>>> ```json
>>>>> {
>>>>> "operators": [
>>>>> {
>>>>> "name": "operator1",
>>>>> "class": "com.datatorrent.lib.operator.Input",
>>>>> "properties": {
>>>>> "property1": "value1"
>>>>> }
>>>>> },
>>>>> {
>>>>> "name": "module1",
>>>>> "class": "com.datatorrent.module.Module1",
>>>>> "properties": {
>>>>> "property1": "value1"
>>>>> }
>>>>> },
>>>>> ],
>>>>> "streams": [
>>>>> {
>>>>> "name": "s1",
>>>>> "source": {
>>>>> "operatorName": "operator1",
>>>>> "portName": "output"
>>>>> },
>>>>> "sinks": [
>>>>> {
>>>>> "operatorName": "module",
>>>>> "portName": "input"
>>>>> }
>>>>> ]
>>>>> }
>>>>> ]
>>>>> }
>>>>> ```
>>>>>
>>>>> In above example "module1" is of type Module. While processing the Json
>>>>>
>>>> if
>>>>
>>>>> the
>>>>> class of component is of type Module then we will use addModule api to
>>>>>
>>>> the
>>>>
>>>>> DAG.
>>>>>
>>>>> Let me know, what do you think about this.
>>>>>
>>>>> Regards,
>>>>> - Tushar.
>>>>>
>>>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <tu...@datatorrent.com>
>>>>> wrote:
>>>>>
>>>>> A module is superset of operator before expansion in logical plan,
>>>>> except
>>>>> operator
>>>>>> attribute are not applicable to modules. As Tim suggested we will have
>>>>>>
>>>>> to
>>>>> add checks
>>>>>> later to validate these attributes settings, and not allow top level
>>>>>> operator and
>>>>>> module to share same name.
>>>>>>
>>>>>> The same question comes when we want to add module to application using
>>>>>> property
>>>>>> file API and JSON file API.
>>>>>>
>>>>>> dt.operator.<operatorname>.classname=<fqcn of operator>
>>>>>>
>>>>>> One approach we can take is, if class is of type module, then use
>>>>>> addModule api
>>>>>> to add module to the dag, else use addOperator api. The initialization
>>>>>>
>>>>> of
>>>>> object and applying property is generic can be reused by module.
>>>>>> Similarly we have Json API where all operators are specified under
>>>>>> operators array
>>>>>>
>>>>>> {
>>>>>> "operators": [
>>>>>> {.. operators .. }
>>>>>> ],
>>>>>> "streams": [
>>>>>> {.. streams .. }
>>>>>> ]
>>>>>> }
>>>>>>
>>>>>> We could add module in the operators array, and based on the object
>>>>>>
>>>>> type
>>>>> we can
>>>>>> consider it as module or operator. The mechanism of initializing object
>>>>>> and applying
>>>>>> properties remains same.
>>>>>>
>>>>>> - Tushar.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
Re: [APEX-107] Support for specifying module properties on modules.
Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
+1. Modules and operators should be interchangeable.
Should something be recommend in terms of naming convention for Module
Class in that case?
- Chinmay.
~ Chinmay.
On Tue, Dec 1, 2015 at 9:51 AM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
> Good point Vlad, if the class implements both Operator and Module
> interface. We will consider it as a module by default in json API, But also
> provide a way to specify his intention whether it should be considered as
> Module or Operator.
>
> If user use a Java API then intention is clear whether the object should be
> considered as Module or Operator (addOperator v/s addModule), will it be
> correct to change the intention of the user internally based on the type of
> component?
>
> - Tushar.
>
>
> On Tue, Dec 1, 2015 at 8:34 AM, Vlad Rozov <v....@datatorrent.com>
> wrote:
>
> > +1. Modules and operators should be interchangeable. It should be
> possible
> > for an operator designer to replace it with a module without requirement
> to
> > recompile all applications that use the existing operator. From
> application
> > designer view both operators and modules may be considered as a black box
> > with a private implementation that may be monolithic (operator) or
> > distributed (module).
> >
> > Note that a Java class may implement both Operator and Module interface
> > and be added to a DAG using addOperator() method. At run-time (Logical
> > Plan->Physical Plan) Apex platform should handle such classes as modules
> > and may not depend on how a module was added to a DAG, IMO.
> >
> > Thank you,
> >
> > Vlad
> >
> >
> > On 11/30/15 08:41, Tushar Gosavi wrote:
> >
> >> addOperator method in DAG interface takes object of type Operator,
> unless
> >> we make Module as subclass of Operator we can not use the same API.
> >>
> >> - Tushar.
> >>
> >>
> >> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <th...@datatorrent.com>
> >> wrote:
> >>
> >> +1
> >>>
> >>> If there are really module specific constructs, they can be added
> later.
> >>>
> >>> Why is the same not supported in the Java API, i.e. why not allow a
> >>> module
> >>> to be added like an operator and handle the distinction in the
> >>> implementation?
> >>>
> >>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tushar@datatorrent.com
> >
> >>> wrote:
> >>>
> >>> Hi All,
> >>>>
> >>>> I am going to add a feature to allow use of modules in property file
> and
> >>>> json api.
> >>>> I am planing to use same "operators" array for adding modules as well
> as
> >>>> operators. The reasoning behind is that user don't have to worry about
> >>>>
> >>> the
> >>>
> >>>> component being added to the DAG is actually an operator or a module,
> as
> >>>> the method to specify their properties and to create instance of them
> is
> >>>> same. And it results in less complex syntax for writing application
> >>>> using
> >>>> json/property file.
> >>>>
> >>>> For example the sample Application specified using Json will look like
> >>>> below.
> >>>> ```json
> >>>> {
> >>>> "operators": [
> >>>> {
> >>>> "name": "operator1",
> >>>> "class": "com.datatorrent.lib.operator.Input",
> >>>> "properties": {
> >>>> "property1": "value1"
> >>>> }
> >>>> },
> >>>> {
> >>>> "name": "module1",
> >>>> "class": "com.datatorrent.module.Module1",
> >>>> "properties": {
> >>>> "property1": "value1"
> >>>> }
> >>>> },
> >>>> ],
> >>>> "streams": [
> >>>> {
> >>>> "name": "s1",
> >>>> "source": {
> >>>> "operatorName": "operator1",
> >>>> "portName": "output"
> >>>> },
> >>>> "sinks": [
> >>>> {
> >>>> "operatorName": "module",
> >>>> "portName": "input"
> >>>> }
> >>>> ]
> >>>> }
> >>>> ]
> >>>> }
> >>>> ```
> >>>>
> >>>> In above example "module1" is of type Module. While processing the
> Json
> >>>>
> >>> if
> >>>
> >>>> the
> >>>> class of component is of type Module then we will use addModule api to
> >>>>
> >>> the
> >>>
> >>>> DAG.
> >>>>
> >>>> Let me know, what do you think about this.
> >>>>
> >>>> Regards,
> >>>> - Tushar.
> >>>>
> >>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <
> tushar@datatorrent.com>
> >>>> wrote:
> >>>>
> >>>> A module is superset of operator before expansion in logical plan,
> >>>>>
> >>>> except
> >>>
> >>>> operator
> >>>>> attribute are not applicable to modules. As Tim suggested we will
> have
> >>>>>
> >>>> to
> >>>
> >>>> add checks
> >>>>> later to validate these attributes settings, and not allow top level
> >>>>> operator and
> >>>>> module to share same name.
> >>>>>
> >>>>> The same question comes when we want to add module to application
> using
> >>>>> property
> >>>>> file API and JSON file API.
> >>>>>
> >>>>> dt.operator.<operatorname>.classname=<fqcn of operator>
> >>>>>
> >>>>> One approach we can take is, if class is of type module, then use
> >>>>> addModule api
> >>>>> to add module to the dag, else use addOperator api. The
> initialization
> >>>>>
> >>>> of
> >>>
> >>>> object and applying property is generic can be reused by module.
> >>>>>
> >>>>> Similarly we have Json API where all operators are specified under
> >>>>> operators array
> >>>>>
> >>>>> {
> >>>>> "operators": [
> >>>>> {.. operators .. }
> >>>>> ],
> >>>>> "streams": [
> >>>>> {.. streams .. }
> >>>>> ]
> >>>>> }
> >>>>>
> >>>>> We could add module in the operators array, and based on the object
> >>>>>
> >>>> type
> >>>
> >>>> we can
> >>>>> consider it as module or operator. The mechanism of initializing
> object
> >>>>> and applying
> >>>>> properties remains same.
> >>>>>
> >>>>> - Tushar.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >
>
Re: [APEX-107] Support for specifying module properties on modules.
Posted by Tushar Gosavi <tu...@datatorrent.com>.
Good point Vlad, if the class implements both Operator and Module
interface. We will consider it as a module by default in json API, But also
provide a way to specify his intention whether it should be considered as
Module or Operator.
If user use a Java API then intention is clear whether the object should be
considered as Module or Operator (addOperator v/s addModule), will it be
correct to change the intention of the user internally based on the type of
component?
- Tushar.
On Tue, Dec 1, 2015 at 8:34 AM, Vlad Rozov <v....@datatorrent.com> wrote:
> +1. Modules and operators should be interchangeable. It should be possible
> for an operator designer to replace it with a module without requirement to
> recompile all applications that use the existing operator. From application
> designer view both operators and modules may be considered as a black box
> with a private implementation that may be monolithic (operator) or
> distributed (module).
>
> Note that a Java class may implement both Operator and Module interface
> and be added to a DAG using addOperator() method. At run-time (Logical
> Plan->Physical Plan) Apex platform should handle such classes as modules
> and may not depend on how a module was added to a DAG, IMO.
>
> Thank you,
>
> Vlad
>
>
> On 11/30/15 08:41, Tushar Gosavi wrote:
>
>> addOperator method in DAG interface takes object of type Operator, unless
>> we make Module as subclass of Operator we can not use the same API.
>>
>> - Tushar.
>>
>>
>> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <th...@datatorrent.com>
>> wrote:
>>
>> +1
>>>
>>> If there are really module specific constructs, they can be added later.
>>>
>>> Why is the same not supported in the Java API, i.e. why not allow a
>>> module
>>> to be added like an operator and handle the distinction in the
>>> implementation?
>>>
>>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tu...@datatorrent.com>
>>> wrote:
>>>
>>> Hi All,
>>>>
>>>> I am going to add a feature to allow use of modules in property file and
>>>> json api.
>>>> I am planing to use same "operators" array for adding modules as well as
>>>> operators. The reasoning behind is that user don't have to worry about
>>>>
>>> the
>>>
>>>> component being added to the DAG is actually an operator or a module, as
>>>> the method to specify their properties and to create instance of them is
>>>> same. And it results in less complex syntax for writing application
>>>> using
>>>> json/property file.
>>>>
>>>> For example the sample Application specified using Json will look like
>>>> below.
>>>> ```json
>>>> {
>>>> "operators": [
>>>> {
>>>> "name": "operator1",
>>>> "class": "com.datatorrent.lib.operator.Input",
>>>> "properties": {
>>>> "property1": "value1"
>>>> }
>>>> },
>>>> {
>>>> "name": "module1",
>>>> "class": "com.datatorrent.module.Module1",
>>>> "properties": {
>>>> "property1": "value1"
>>>> }
>>>> },
>>>> ],
>>>> "streams": [
>>>> {
>>>> "name": "s1",
>>>> "source": {
>>>> "operatorName": "operator1",
>>>> "portName": "output"
>>>> },
>>>> "sinks": [
>>>> {
>>>> "operatorName": "module",
>>>> "portName": "input"
>>>> }
>>>> ]
>>>> }
>>>> ]
>>>> }
>>>> ```
>>>>
>>>> In above example "module1" is of type Module. While processing the Json
>>>>
>>> if
>>>
>>>> the
>>>> class of component is of type Module then we will use addModule api to
>>>>
>>> the
>>>
>>>> DAG.
>>>>
>>>> Let me know, what do you think about this.
>>>>
>>>> Regards,
>>>> - Tushar.
>>>>
>>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <tu...@datatorrent.com>
>>>> wrote:
>>>>
>>>> A module is superset of operator before expansion in logical plan,
>>>>>
>>>> except
>>>
>>>> operator
>>>>> attribute are not applicable to modules. As Tim suggested we will have
>>>>>
>>>> to
>>>
>>>> add checks
>>>>> later to validate these attributes settings, and not allow top level
>>>>> operator and
>>>>> module to share same name.
>>>>>
>>>>> The same question comes when we want to add module to application using
>>>>> property
>>>>> file API and JSON file API.
>>>>>
>>>>> dt.operator.<operatorname>.classname=<fqcn of operator>
>>>>>
>>>>> One approach we can take is, if class is of type module, then use
>>>>> addModule api
>>>>> to add module to the dag, else use addOperator api. The initialization
>>>>>
>>>> of
>>>
>>>> object and applying property is generic can be reused by module.
>>>>>
>>>>> Similarly we have Json API where all operators are specified under
>>>>> operators array
>>>>>
>>>>> {
>>>>> "operators": [
>>>>> {.. operators .. }
>>>>> ],
>>>>> "streams": [
>>>>> {.. streams .. }
>>>>> ]
>>>>> }
>>>>>
>>>>> We could add module in the operators array, and based on the object
>>>>>
>>>> type
>>>
>>>> we can
>>>>> consider it as module or operator. The mechanism of initializing object
>>>>> and applying
>>>>> properties remains same.
>>>>>
>>>>> - Tushar.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>
Re: [APEX-107] Support for specifying module properties on modules.
Posted by Vlad Rozov <v....@datatorrent.com>.
+1. Modules and operators should be interchangeable. It should be
possible for an operator designer to replace it with a module without
requirement to recompile all applications that use the existing
operator. From application designer view both operators and modules may
be considered as a black box with a private implementation that may be
monolithic (operator) or distributed (module).
Note that a Java class may implement both Operator and Module interface
and be added to a DAG using addOperator() method. At run-time (Logical
Plan->Physical Plan) Apex platform should handle such classes as modules
and may not depend on how a module was added to a DAG, IMO.
Thank you,
Vlad
On 11/30/15 08:41, Tushar Gosavi wrote:
> addOperator method in DAG interface takes object of type Operator, unless
> we make Module as subclass of Operator we can not use the same API.
>
> - Tushar.
>
>
> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <th...@datatorrent.com>
> wrote:
>
>> +1
>>
>> If there are really module specific constructs, they can be added later.
>>
>> Why is the same not supported in the Java API, i.e. why not allow a module
>> to be added like an operator and handle the distinction in the
>> implementation?
>>
>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tu...@datatorrent.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> I am going to add a feature to allow use of modules in property file and
>>> json api.
>>> I am planing to use same "operators" array for adding modules as well as
>>> operators. The reasoning behind is that user don't have to worry about
>> the
>>> component being added to the DAG is actually an operator or a module, as
>>> the method to specify their properties and to create instance of them is
>>> same. And it results in less complex syntax for writing application using
>>> json/property file.
>>>
>>> For example the sample Application specified using Json will look like
>>> below.
>>> ```json
>>> {
>>> "operators": [
>>> {
>>> "name": "operator1",
>>> "class": "com.datatorrent.lib.operator.Input",
>>> "properties": {
>>> "property1": "value1"
>>> }
>>> },
>>> {
>>> "name": "module1",
>>> "class": "com.datatorrent.module.Module1",
>>> "properties": {
>>> "property1": "value1"
>>> }
>>> },
>>> ],
>>> "streams": [
>>> {
>>> "name": "s1",
>>> "source": {
>>> "operatorName": "operator1",
>>> "portName": "output"
>>> },
>>> "sinks": [
>>> {
>>> "operatorName": "module",
>>> "portName": "input"
>>> }
>>> ]
>>> }
>>> ]
>>> }
>>> ```
>>>
>>> In above example "module1" is of type Module. While processing the Json
>> if
>>> the
>>> class of component is of type Module then we will use addModule api to
>> the
>>> DAG.
>>>
>>> Let me know, what do you think about this.
>>>
>>> Regards,
>>> - Tushar.
>>>
>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <tu...@datatorrent.com>
>>> wrote:
>>>
>>>> A module is superset of operator before expansion in logical plan,
>> except
>>>> operator
>>>> attribute are not applicable to modules. As Tim suggested we will have
>> to
>>>> add checks
>>>> later to validate these attributes settings, and not allow top level
>>>> operator and
>>>> module to share same name.
>>>>
>>>> The same question comes when we want to add module to application using
>>>> property
>>>> file API and JSON file API.
>>>>
>>>> dt.operator.<operatorname>.classname=<fqcn of operator>
>>>>
>>>> One approach we can take is, if class is of type module, then use
>>>> addModule api
>>>> to add module to the dag, else use addOperator api. The initialization
>> of
>>>> object and applying property is generic can be reused by module.
>>>>
>>>> Similarly we have Json API where all operators are specified under
>>>> operators array
>>>>
>>>> {
>>>> "operators": [
>>>> {.. operators .. }
>>>> ],
>>>> "streams": [
>>>> {.. streams .. }
>>>> ]
>>>> }
>>>>
>>>> We could add module in the operators array, and based on the object
>> type
>>>> we can
>>>> consider it as module or operator. The mechanism of initializing object
>>>> and applying
>>>> properties remains same.
>>>>
>>>> - Tushar.
>>>>
>>>>
>>>>
>>>>
>>>>
Re: [APEX-107] Support for specifying module properties on modules.
Posted by Tushar Gosavi <tu...@datatorrent.com>.
addOperator method in DAG interface takes object of type Operator, unless
we make Module as subclass of Operator we can not use the same API.
- Tushar.
On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <th...@datatorrent.com>
wrote:
> +1
>
> If there are really module specific constructs, they can be added later.
>
> Why is the same not supported in the Java API, i.e. why not allow a module
> to be added like an operator and handle the distinction in the
> implementation?
>
> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
> > Hi All,
> >
> > I am going to add a feature to allow use of modules in property file and
> > json api.
> > I am planing to use same "operators" array for adding modules as well as
> > operators. The reasoning behind is that user don't have to worry about
> the
> > component being added to the DAG is actually an operator or a module, as
> > the method to specify their properties and to create instance of them is
> > same. And it results in less complex syntax for writing application using
> > json/property file.
> >
> > For example the sample Application specified using Json will look like
> > below.
> > ```json
> > {
> > "operators": [
> > {
> > "name": "operator1",
> > "class": "com.datatorrent.lib.operator.Input",
> > "properties": {
> > "property1": "value1"
> > }
> > },
> > {
> > "name": "module1",
> > "class": "com.datatorrent.module.Module1",
> > "properties": {
> > "property1": "value1"
> > }
> > },
> > ],
> > "streams": [
> > {
> > "name": "s1",
> > "source": {
> > "operatorName": "operator1",
> > "portName": "output"
> > },
> > "sinks": [
> > {
> > "operatorName": "module",
> > "portName": "input"
> > }
> > ]
> > }
> > ]
> > }
> > ```
> >
> > In above example "module1" is of type Module. While processing the Json
> if
> > the
> > class of component is of type Module then we will use addModule api to
> the
> > DAG.
> >
> > Let me know, what do you think about this.
> >
> > Regards,
> > - Tushar.
> >
> > On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <tu...@datatorrent.com>
> > wrote:
> >
> > >
> > > A module is superset of operator before expansion in logical plan,
> except
> > > operator
> > > attribute are not applicable to modules. As Tim suggested we will have
> to
> > > add checks
> > > later to validate these attributes settings, and not allow top level
> > > operator and
> > > module to share same name.
> > >
> > > The same question comes when we want to add module to application using
> > > property
> > > file API and JSON file API.
> > >
> > > dt.operator.<operatorname>.classname=<fqcn of operator>
> > >
> > > One approach we can take is, if class is of type module, then use
> > > addModule api
> > > to add module to the dag, else use addOperator api. The initialization
> of
> > > object and applying property is generic can be reused by module.
> > >
> > > Similarly we have Json API where all operators are specified under
> > > operators array
> > >
> > > {
> > > "operators": [
> > > {.. operators .. }
> > > ],
> > > "streams": [
> > > {.. streams .. }
> > > ]
> > > }
> > >
> > > We could add module in the operators array, and based on the object
> type
> > > we can
> > > consider it as module or operator. The mechanism of initializing object
> > > and applying
> > > properties remains same.
> > >
> > > - Tushar.
> > >
> > >
> > >
> > >
> > >
> >
>
Re: [APEX-107] Support for specifying module properties on modules.
Posted by Tushar Gosavi <tu...@datatorrent.com>.
It will be possible to connect output port to input port of same module
using
old json format. The DAG validation is performed post expansion. Another
advantage of keeping json format compatible with the existing format is
that it
will allow us to use existing tools (dtAssemble) with little/no changes.
On Tue, Dec 1, 2015 at 1:01 AM, David Yan <da...@datatorrent.com> wrote:
> I also prefer an explicit modules section. The user of the module needs to
> know it's a module, and if my understanding of modules is still correct,
> modules are not operators because it may be valid for a module's output
> port to connect to its own input port.
>
> David
>
> On Mon, Nov 30, 2015 at 8:40 AM, Siyuan Hua <si...@datatorrent.com>
> wrote:
>
> > I prefer explicit "modules" section. Just because it's more intuitive.
> >
> > On Mon, Nov 30, 2015 at 8:21 AM, Thomas Weise <th...@datatorrent.com>
> > wrote:
> >
> > > +1
> > >
> > > If there are really module specific constructs, they can be added
> later.
> > >
> > > Why is the same not supported in the Java API, i.e. why not allow a
> > module
> > > to be added like an operator and handle the distinction in the
> > > implementation?
> > >
> > > On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tushar@datatorrent.com
> >
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I am going to add a feature to allow use of modules in property file
> > and
> > > > json api.
> > > > I am planing to use same "operators" array for adding modules as well
> > as
> > > > operators. The reasoning behind is that user don't have to worry
> about
> > > the
> > > > component being added to the DAG is actually an operator or a module,
> > as
> > > > the method to specify their properties and to create instance of them
> > is
> > > > same. And it results in less complex syntax for writing application
> > using
> > > > json/property file.
> > > >
> > > > For example the sample Application specified using Json will look
> like
> > > > below.
> > > > ```json
> > > > {
> > > > "operators": [
> > > > {
> > > > "name": "operator1",
> > > > "class": "com.datatorrent.lib.operator.Input",
> > > > "properties": {
> > > > "property1": "value1"
> > > > }
> > > > },
> > > > {
> > > > "name": "module1",
> > > > "class": "com.datatorrent.module.Module1",
> > > > "properties": {
> > > > "property1": "value1"
> > > > }
> > > > },
> > > > ],
> > > > "streams": [
> > > > {
> > > > "name": "s1",
> > > > "source": {
> > > > "operatorName": "operator1",
> > > > "portName": "output"
> > > > },
> > > > "sinks": [
> > > > {
> > > > "operatorName": "module",
> > > > "portName": "input"
> > > > }
> > > > ]
> > > > }
> > > > ]
> > > > }
> > > > ```
> > > >
> > > > In above example "module1" is of type Module. While processing the
> Json
> > > if
> > > > the
> > > > class of component is of type Module then we will use addModule api
> to
> > > the
> > > > DAG.
> > > >
> > > > Let me know, what do you think about this.
> > > >
> > > > Regards,
> > > > - Tushar.
> > > >
> > > > On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <
> tushar@datatorrent.com
> > >
> > > > wrote:
> > > >
> > > > >
> > > > > A module is superset of operator before expansion in logical plan,
> > > except
> > > > > operator
> > > > > attribute are not applicable to modules. As Tim suggested we will
> > have
> > > to
> > > > > add checks
> > > > > later to validate these attributes settings, and not allow top
> level
> > > > > operator and
> > > > > module to share same name.
> > > > >
> > > > > The same question comes when we want to add module to application
> > using
> > > > > property
> > > > > file API and JSON file API.
> > > > >
> > > > > dt.operator.<operatorname>.classname=<fqcn of operator>
> > > > >
> > > > > One approach we can take is, if class is of type module, then use
> > > > > addModule api
> > > > > to add module to the dag, else use addOperator api. The
> > initialization
> > > of
> > > > > object and applying property is generic can be reused by module.
> > > > >
> > > > > Similarly we have Json API where all operators are specified under
> > > > > operators array
> > > > >
> > > > > {
> > > > > "operators": [
> > > > > {.. operators .. }
> > > > > ],
> > > > > "streams": [
> > > > > {.. streams .. }
> > > > > ]
> > > > > }
> > > > >
> > > > > We could add module in the operators array, and based on the object
> > > type
> > > > > we can
> > > > > consider it as module or operator. The mechanism of initializing
> > object
> > > > > and applying
> > > > > properties remains same.
> > > > >
> > > > > - Tushar.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [APEX-107] Support for specifying module properties on modules.
Posted by David Yan <da...@datatorrent.com>.
I also prefer an explicit modules section. The user of the module needs to
know it's a module, and if my understanding of modules is still correct,
modules are not operators because it may be valid for a module's output
port to connect to its own input port.
David
On Mon, Nov 30, 2015 at 8:40 AM, Siyuan Hua <si...@datatorrent.com> wrote:
> I prefer explicit "modules" section. Just because it's more intuitive.
>
> On Mon, Nov 30, 2015 at 8:21 AM, Thomas Weise <th...@datatorrent.com>
> wrote:
>
> > +1
> >
> > If there are really module specific constructs, they can be added later.
> >
> > Why is the same not supported in the Java API, i.e. why not allow a
> module
> > to be added like an operator and handle the distinction in the
> > implementation?
> >
> > On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tu...@datatorrent.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I am going to add a feature to allow use of modules in property file
> and
> > > json api.
> > > I am planing to use same "operators" array for adding modules as well
> as
> > > operators. The reasoning behind is that user don't have to worry about
> > the
> > > component being added to the DAG is actually an operator or a module,
> as
> > > the method to specify their properties and to create instance of them
> is
> > > same. And it results in less complex syntax for writing application
> using
> > > json/property file.
> > >
> > > For example the sample Application specified using Json will look like
> > > below.
> > > ```json
> > > {
> > > "operators": [
> > > {
> > > "name": "operator1",
> > > "class": "com.datatorrent.lib.operator.Input",
> > > "properties": {
> > > "property1": "value1"
> > > }
> > > },
> > > {
> > > "name": "module1",
> > > "class": "com.datatorrent.module.Module1",
> > > "properties": {
> > > "property1": "value1"
> > > }
> > > },
> > > ],
> > > "streams": [
> > > {
> > > "name": "s1",
> > > "source": {
> > > "operatorName": "operator1",
> > > "portName": "output"
> > > },
> > > "sinks": [
> > > {
> > > "operatorName": "module",
> > > "portName": "input"
> > > }
> > > ]
> > > }
> > > ]
> > > }
> > > ```
> > >
> > > In above example "module1" is of type Module. While processing the Json
> > if
> > > the
> > > class of component is of type Module then we will use addModule api to
> > the
> > > DAG.
> > >
> > > Let me know, what do you think about this.
> > >
> > > Regards,
> > > - Tushar.
> > >
> > > On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <tushar@datatorrent.com
> >
> > > wrote:
> > >
> > > >
> > > > A module is superset of operator before expansion in logical plan,
> > except
> > > > operator
> > > > attribute are not applicable to modules. As Tim suggested we will
> have
> > to
> > > > add checks
> > > > later to validate these attributes settings, and not allow top level
> > > > operator and
> > > > module to share same name.
> > > >
> > > > The same question comes when we want to add module to application
> using
> > > > property
> > > > file API and JSON file API.
> > > >
> > > > dt.operator.<operatorname>.classname=<fqcn of operator>
> > > >
> > > > One approach we can take is, if class is of type module, then use
> > > > addModule api
> > > > to add module to the dag, else use addOperator api. The
> initialization
> > of
> > > > object and applying property is generic can be reused by module.
> > > >
> > > > Similarly we have Json API where all operators are specified under
> > > > operators array
> > > >
> > > > {
> > > > "operators": [
> > > > {.. operators .. }
> > > > ],
> > > > "streams": [
> > > > {.. streams .. }
> > > > ]
> > > > }
> > > >
> > > > We could add module in the operators array, and based on the object
> > type
> > > > we can
> > > > consider it as module or operator. The mechanism of initializing
> object
> > > > and applying
> > > > properties remains same.
> > > >
> > > > - Tushar.
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: [APEX-107] Support for specifying module properties on modules.
Posted by Siyuan Hua <si...@datatorrent.com>.
I prefer explicit "modules" section. Just because it's more intuitive.
On Mon, Nov 30, 2015 at 8:21 AM, Thomas Weise <th...@datatorrent.com>
wrote:
> +1
>
> If there are really module specific constructs, they can be added later.
>
> Why is the same not supported in the Java API, i.e. why not allow a module
> to be added like an operator and handle the distinction in the
> implementation?
>
> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
> > Hi All,
> >
> > I am going to add a feature to allow use of modules in property file and
> > json api.
> > I am planing to use same "operators" array for adding modules as well as
> > operators. The reasoning behind is that user don't have to worry about
> the
> > component being added to the DAG is actually an operator or a module, as
> > the method to specify their properties and to create instance of them is
> > same. And it results in less complex syntax for writing application using
> > json/property file.
> >
> > For example the sample Application specified using Json will look like
> > below.
> > ```json
> > {
> > "operators": [
> > {
> > "name": "operator1",
> > "class": "com.datatorrent.lib.operator.Input",
> > "properties": {
> > "property1": "value1"
> > }
> > },
> > {
> > "name": "module1",
> > "class": "com.datatorrent.module.Module1",
> > "properties": {
> > "property1": "value1"
> > }
> > },
> > ],
> > "streams": [
> > {
> > "name": "s1",
> > "source": {
> > "operatorName": "operator1",
> > "portName": "output"
> > },
> > "sinks": [
> > {
> > "operatorName": "module",
> > "portName": "input"
> > }
> > ]
> > }
> > ]
> > }
> > ```
> >
> > In above example "module1" is of type Module. While processing the Json
> if
> > the
> > class of component is of type Module then we will use addModule api to
> the
> > DAG.
> >
> > Let me know, what do you think about this.
> >
> > Regards,
> > - Tushar.
> >
> > On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <tu...@datatorrent.com>
> > wrote:
> >
> > >
> > > A module is superset of operator before expansion in logical plan,
> except
> > > operator
> > > attribute are not applicable to modules. As Tim suggested we will have
> to
> > > add checks
> > > later to validate these attributes settings, and not allow top level
> > > operator and
> > > module to share same name.
> > >
> > > The same question comes when we want to add module to application using
> > > property
> > > file API and JSON file API.
> > >
> > > dt.operator.<operatorname>.classname=<fqcn of operator>
> > >
> > > One approach we can take is, if class is of type module, then use
> > > addModule api
> > > to add module to the dag, else use addOperator api. The initialization
> of
> > > object and applying property is generic can be reused by module.
> > >
> > > Similarly we have Json API where all operators are specified under
> > > operators array
> > >
> > > {
> > > "operators": [
> > > {.. operators .. }
> > > ],
> > > "streams": [
> > > {.. streams .. }
> > > ]
> > > }
> > >
> > > We could add module in the operators array, and based on the object
> type
> > > we can
> > > consider it as module or operator. The mechanism of initializing object
> > > and applying
> > > properties remains same.
> > >
> > > - Tushar.
> > >
> > >
> > >
> > >
> > >
> >
>
Re: [APEX-107] Support for specifying module properties on modules.
Posted by Thomas Weise <th...@datatorrent.com>.
+1
If there are really module specific constructs, they can be added later.
Why is the same not supported in the Java API, i.e. why not allow a module
to be added like an operator and handle the distinction in the
implementation?
On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
> Hi All,
>
> I am going to add a feature to allow use of modules in property file and
> json api.
> I am planing to use same "operators" array for adding modules as well as
> operators. The reasoning behind is that user don't have to worry about the
> component being added to the DAG is actually an operator or a module, as
> the method to specify their properties and to create instance of them is
> same. And it results in less complex syntax for writing application using
> json/property file.
>
> For example the sample Application specified using Json will look like
> below.
> ```json
> {
> "operators": [
> {
> "name": "operator1",
> "class": "com.datatorrent.lib.operator.Input",
> "properties": {
> "property1": "value1"
> }
> },
> {
> "name": "module1",
> "class": "com.datatorrent.module.Module1",
> "properties": {
> "property1": "value1"
> }
> },
> ],
> "streams": [
> {
> "name": "s1",
> "source": {
> "operatorName": "operator1",
> "portName": "output"
> },
> "sinks": [
> {
> "operatorName": "module",
> "portName": "input"
> }
> ]
> }
> ]
> }
> ```
>
> In above example "module1" is of type Module. While processing the Json if
> the
> class of component is of type Module then we will use addModule api to the
> DAG.
>
> Let me know, what do you think about this.
>
> Regards,
> - Tushar.
>
> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
> >
> > A module is superset of operator before expansion in logical plan, except
> > operator
> > attribute are not applicable to modules. As Tim suggested we will have to
> > add checks
> > later to validate these attributes settings, and not allow top level
> > operator and
> > module to share same name.
> >
> > The same question comes when we want to add module to application using
> > property
> > file API and JSON file API.
> >
> > dt.operator.<operatorname>.classname=<fqcn of operator>
> >
> > One approach we can take is, if class is of type module, then use
> > addModule api
> > to add module to the dag, else use addOperator api. The initialization of
> > object and applying property is generic can be reused by module.
> >
> > Similarly we have Json API where all operators are specified under
> > operators array
> >
> > {
> > "operators": [
> > {.. operators .. }
> > ],
> > "streams": [
> > {.. streams .. }
> > ]
> > }
> >
> > We could add module in the operators array, and based on the object type
> > we can
> > consider it as module or operator. The mechanism of initializing object
> > and applying
> > properties remains same.
> >
> > - Tushar.
> >
> >
> >
> >
> >
>