You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Vlad Rozov <v....@datatorrent.com> on 2015/12/01 04:04:12 UTC
Re: [APEX-107] Support for specifying module properties on modules.
+1. Modules and operators should be interchangeable. It should be
possible for an operator designer to replace it with a module without
requirement to recompile all applications that use the existing
operator. From application designer view both operators and modules may
be considered as a black box with a private implementation that may be
monolithic (operator) or distributed (module).
Note that a Java class may implement both Operator and Module interface
and be added to a DAG using addOperator() method. At run-time (Logical
Plan->Physical Plan) Apex platform should handle such classes as modules
and may not depend on how a module was added to a DAG, IMO.
Thank you,
Vlad
On 11/30/15 08:41, Tushar Gosavi wrote:
> addOperator method in DAG interface takes object of type Operator, unless
> we make Module as subclass of Operator we can not use the same API.
>
> - Tushar.
>
>
> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <th...@datatorrent.com>
> wrote:
>
>> +1
>>
>> If there are really module specific constructs, they can be added later.
>>
>> Why is the same not supported in the Java API, i.e. why not allow a module
>> to be added like an operator and handle the distinction in the
>> implementation?
>>
>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tu...@datatorrent.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> I am going to add a feature to allow use of modules in property file and
>>> json api.
>>> I am planing to use same "operators" array for adding modules as well as
>>> operators. The reasoning behind is that user don't have to worry about
>> the
>>> component being added to the DAG is actually an operator or a module, as
>>> the method to specify their properties and to create instance of them is
>>> same. And it results in less complex syntax for writing application using
>>> json/property file.
>>>
>>> For example the sample Application specified using Json will look like
>>> below.
>>> ```json
>>> {
>>> "operators": [
>>> {
>>> "name": "operator1",
>>> "class": "com.datatorrent.lib.operator.Input",
>>> "properties": {
>>> "property1": "value1"
>>> }
>>> },
>>> {
>>> "name": "module1",
>>> "class": "com.datatorrent.module.Module1",
>>> "properties": {
>>> "property1": "value1"
>>> }
>>> },
>>> ],
>>> "streams": [
>>> {
>>> "name": "s1",
>>> "source": {
>>> "operatorName": "operator1",
>>> "portName": "output"
>>> },
>>> "sinks": [
>>> {
>>> "operatorName": "module",
>>> "portName": "input"
>>> }
>>> ]
>>> }
>>> ]
>>> }
>>> ```
>>>
>>> In above example "module1" is of type Module. While processing the Json
>> if
>>> the
>>> class of component is of type Module then we will use addModule api to
>> the
>>> DAG.
>>>
>>> Let me know, what do you think about this.
>>>
>>> Regards,
>>> - Tushar.
>>>
>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <tu...@datatorrent.com>
>>> wrote:
>>>
>>>> A module is superset of operator before expansion in logical plan,
>> except
>>>> operator
>>>> attribute are not applicable to modules. As Tim suggested we will have
>> to
>>>> add checks
>>>> later to validate these attributes settings, and not allow top level
>>>> operator and
>>>> module to share same name.
>>>>
>>>> The same question comes when we want to add module to application using
>>>> property
>>>> file API and JSON file API.
>>>>
>>>> dt.operator.<operatorname>.classname=<fqcn of operator>
>>>>
>>>> One approach we can take is, if class is of type module, then use
>>>> addModule api
>>>> to add module to the dag, else use addOperator api. The initialization
>> of
>>>> object and applying property is generic can be reused by module.
>>>>
>>>> Similarly we have Json API where all operators are specified under
>>>> operators array
>>>>
>>>> {
>>>> "operators": [
>>>> {.. operators .. }
>>>> ],
>>>> "streams": [
>>>> {.. streams .. }
>>>> ]
>>>> }
>>>>
>>>> We could add module in the operators array, and based on the object
>> type
>>>> we can
>>>> consider it as module or operator. The mechanism of initializing object
>>>> and applying
>>>> properties remains same.
>>>>
>>>> - Tushar.
>>>>
>>>>
>>>>
>>>>
>>>>
Re: [APEX-107] Support for specifying module properties on modules.
Posted by Sandeep Deshmukh <sa...@datatorrent.com>.
+1 for modules and operators to be used interchangeably.
Regards,
Sandeep
On Tue, Dec 1, 2015 at 11:54 PM, Vlad Rozov <v....@datatorrent.com> wrote:
> We should either provide Java API that supports interchangeability between
> operators and modules or ignore how an operator/module was added to a DAG.
>
> Thank you,
>
> Vlad
>
>
> On 11/30/15 20:21, Tushar Gosavi wrote:
>
>> Good point Vlad, if the class implements both Operator and Module
>> interface. We will consider it as a module by default in json API, But
>> also
>> provide a way to specify his intention whether it should be considered as
>> Module or Operator.
>>
>> If user use a Java API then intention is clear whether the object should
>> be
>> considered as Module or Operator (addOperator v/s addModule), will it be
>> correct to change the intention of the user internally based on the type
>> of
>> component?
>>
>> - Tushar.
>>
>>
>> On Tue, Dec 1, 2015 at 8:34 AM, Vlad Rozov <v....@datatorrent.com>
>> wrote:
>>
>> +1. Modules and operators should be interchangeable. It should be possible
>>> for an operator designer to replace it with a module without requirement
>>> to
>>> recompile all applications that use the existing operator. From
>>> application
>>> designer view both operators and modules may be considered as a black box
>>> with a private implementation that may be monolithic (operator) or
>>> distributed (module).
>>>
>>> Note that a Java class may implement both Operator and Module interface
>>> and be added to a DAG using addOperator() method. At run-time (Logical
>>> Plan->Physical Plan) Apex platform should handle such classes as modules
>>> and may not depend on how a module was added to a DAG, IMO.
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>>
>>> On 11/30/15 08:41, Tushar Gosavi wrote:
>>>
>>> addOperator method in DAG interface takes object of type Operator, unless
>>>> we make Module as subclass of Operator we can not use the same API.
>>>>
>>>> - Tushar.
>>>>
>>>>
>>>> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <th...@datatorrent.com>
>>>> wrote:
>>>>
>>>> +1
>>>>
>>>>> If there are really module specific constructs, they can be added
>>>>> later.
>>>>>
>>>>> Why is the same not supported in the Java API, i.e. why not allow a
>>>>> module
>>>>> to be added like an operator and handle the distinction in the
>>>>> implementation?
>>>>>
>>>>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tushar@datatorrent.com
>>>>> >
>>>>> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>>> I am going to add a feature to allow use of modules in property file
>>>>>> and
>>>>>> json api.
>>>>>> I am planing to use same "operators" array for adding modules as well
>>>>>> as
>>>>>> operators. The reasoning behind is that user don't have to worry about
>>>>>>
>>>>>> the
>>>>>
>>>>> component being added to the DAG is actually an operator or a module,
>>>>>> as
>>>>>> the method to specify their properties and to create instance of them
>>>>>> is
>>>>>> same. And it results in less complex syntax for writing application
>>>>>> using
>>>>>> json/property file.
>>>>>>
>>>>>> For example the sample Application specified using Json will look like
>>>>>> below.
>>>>>> ```json
>>>>>> {
>>>>>> "operators": [
>>>>>> {
>>>>>> "name": "operator1",
>>>>>> "class": "com.datatorrent.lib.operator.Input",
>>>>>> "properties": {
>>>>>> "property1": "value1"
>>>>>> }
>>>>>> },
>>>>>> {
>>>>>> "name": "module1",
>>>>>> "class": "com.datatorrent.module.Module1",
>>>>>> "properties": {
>>>>>> "property1": "value1"
>>>>>> }
>>>>>> },
>>>>>> ],
>>>>>> "streams": [
>>>>>> {
>>>>>> "name": "s1",
>>>>>> "source": {
>>>>>> "operatorName": "operator1",
>>>>>> "portName": "output"
>>>>>> },
>>>>>> "sinks": [
>>>>>> {
>>>>>> "operatorName": "module",
>>>>>> "portName": "input"
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> ```
>>>>>>
>>>>>> In above example "module1" is of type Module. While processing the
>>>>>> Json
>>>>>>
>>>>>> if
>>>>>
>>>>> the
>>>>>> class of component is of type Module then we will use addModule api to
>>>>>>
>>>>>> the
>>>>>
>>>>> DAG.
>>>>>>
>>>>>> Let me know, what do you think about this.
>>>>>>
>>>>>> Regards,
>>>>>> - Tushar.
>>>>>>
>>>>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <
>>>>>> tushar@datatorrent.com>
>>>>>> wrote:
>>>>>>
>>>>>> A module is superset of operator before expansion in logical plan,
>>>>>> except
>>>>>> operator
>>>>>>
>>>>>>> attribute are not applicable to modules. As Tim suggested we will
>>>>>>> have
>>>>>>>
>>>>>>> to
>>>>>> add checks
>>>>>>
>>>>>>> later to validate these attributes settings, and not allow top level
>>>>>>> operator and
>>>>>>> module to share same name.
>>>>>>>
>>>>>>> The same question comes when we want to add module to application
>>>>>>> using
>>>>>>> property
>>>>>>> file API and JSON file API.
>>>>>>>
>>>>>>> dt.operator.<operatorname>.classname=<fqcn of operator>
>>>>>>>
>>>>>>> One approach we can take is, if class is of type module, then use
>>>>>>> addModule api
>>>>>>> to add module to the dag, else use addOperator api. The
>>>>>>> initialization
>>>>>>>
>>>>>>> of
>>>>>> object and applying property is generic can be reused by module.
>>>>>>
>>>>>>> Similarly we have Json API where all operators are specified under
>>>>>>> operators array
>>>>>>>
>>>>>>> {
>>>>>>> "operators": [
>>>>>>> {.. operators .. }
>>>>>>> ],
>>>>>>> "streams": [
>>>>>>> {.. streams .. }
>>>>>>> ]
>>>>>>> }
>>>>>>>
>>>>>>> We could add module in the operators array, and based on the object
>>>>>>>
>>>>>>> type
>>>>>> we can
>>>>>>
>>>>>>> consider it as module or operator. The mechanism of initializing
>>>>>>> object
>>>>>>> and applying
>>>>>>> properties remains same.
>>>>>>>
>>>>>>> - Tushar.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>
Re: [APEX-107] Support for specifying module properties on modules.
Posted by Tushar Gosavi <tu...@datatorrent.com>.
Thanks Vlad, I will check if we can override addOperator in DAG to accept
Module too. This way we can have similar API.
- Tushar.
On Tue, Dec 1, 2015 at 11:54 PM, Vlad Rozov <v....@datatorrent.com> wrote:
> We should either provide Java API that supports interchangeability between
> operators and modules or ignore how an operator/module was added to a DAG.
>
> Thank you,
>
> Vlad
>
>
> On 11/30/15 20:21, Tushar Gosavi wrote:
>
>> Good point Vlad, if the class implements both Operator and Module
>> interface. We will consider it as a module by default in json API, But
>> also
>> provide a way to specify his intention whether it should be considered as
>> Module or Operator.
>>
>> If user use a Java API then intention is clear whether the object should
>> be
>> considered as Module or Operator (addOperator v/s addModule), will it be
>> correct to change the intention of the user internally based on the type
>> of
>> component?
>>
>> - Tushar.
>>
>>
>> On Tue, Dec 1, 2015 at 8:34 AM, Vlad Rozov <v....@datatorrent.com>
>> wrote:
>>
>> +1. Modules and operators should be interchangeable. It should be possible
>>> for an operator designer to replace it with a module without requirement
>>> to
>>> recompile all applications that use the existing operator. From
>>> application
>>> designer view both operators and modules may be considered as a black box
>>> with a private implementation that may be monolithic (operator) or
>>> distributed (module).
>>>
>>> Note that a Java class may implement both Operator and Module interface
>>> and be added to a DAG using addOperator() method. At run-time (Logical
>>> Plan->Physical Plan) Apex platform should handle such classes as modules
>>> and may not depend on how a module was added to a DAG, IMO.
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>>
>>> On 11/30/15 08:41, Tushar Gosavi wrote:
>>>
>>> addOperator method in DAG interface takes object of type Operator, unless
>>>> we make Module as subclass of Operator we can not use the same API.
>>>>
>>>> - Tushar.
>>>>
>>>>
>>>> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <th...@datatorrent.com>
>>>> wrote:
>>>>
>>>> +1
>>>>
>>>>> If there are really module specific constructs, they can be added
>>>>> later.
>>>>>
>>>>> Why is the same not supported in the Java API, i.e. why not allow a
>>>>> module
>>>>> to be added like an operator and handle the distinction in the
>>>>> implementation?
>>>>>
>>>>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tushar@datatorrent.com
>>>>> >
>>>>> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>>> I am going to add a feature to allow use of modules in property file
>>>>>> and
>>>>>> json api.
>>>>>> I am planing to use same "operators" array for adding modules as well
>>>>>> as
>>>>>> operators. The reasoning behind is that user don't have to worry about
>>>>>>
>>>>>> the
>>>>>
>>>>> component being added to the DAG is actually an operator or a module,
>>>>>> as
>>>>>> the method to specify their properties and to create instance of them
>>>>>> is
>>>>>> same. And it results in less complex syntax for writing application
>>>>>> using
>>>>>> json/property file.
>>>>>>
>>>>>> For example the sample Application specified using Json will look like
>>>>>> below.
>>>>>> ```json
>>>>>> {
>>>>>> "operators": [
>>>>>> {
>>>>>> "name": "operator1",
>>>>>> "class": "com.datatorrent.lib.operator.Input",
>>>>>> "properties": {
>>>>>> "property1": "value1"
>>>>>> }
>>>>>> },
>>>>>> {
>>>>>> "name": "module1",
>>>>>> "class": "com.datatorrent.module.Module1",
>>>>>> "properties": {
>>>>>> "property1": "value1"
>>>>>> }
>>>>>> },
>>>>>> ],
>>>>>> "streams": [
>>>>>> {
>>>>>> "name": "s1",
>>>>>> "source": {
>>>>>> "operatorName": "operator1",
>>>>>> "portName": "output"
>>>>>> },
>>>>>> "sinks": [
>>>>>> {
>>>>>> "operatorName": "module",
>>>>>> "portName": "input"
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> ```
>>>>>>
>>>>>> In above example "module1" is of type Module. While processing the
>>>>>> Json
>>>>>>
>>>>>> if
>>>>>
>>>>> the
>>>>>> class of component is of type Module then we will use addModule api to
>>>>>>
>>>>>> the
>>>>>
>>>>> DAG.
>>>>>>
>>>>>> Let me know, what do you think about this.
>>>>>>
>>>>>> Regards,
>>>>>> - Tushar.
>>>>>>
>>>>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <
>>>>>> tushar@datatorrent.com>
>>>>>> wrote:
>>>>>>
>>>>>> A module is superset of operator before expansion in logical plan,
>>>>>> except
>>>>>> operator
>>>>>>
>>>>>>> attribute are not applicable to modules. As Tim suggested we will
>>>>>>> have
>>>>>>>
>>>>>>> to
>>>>>> add checks
>>>>>>
>>>>>>> later to validate these attributes settings, and not allow top level
>>>>>>> operator and
>>>>>>> module to share same name.
>>>>>>>
>>>>>>> The same question comes when we want to add module to application
>>>>>>> using
>>>>>>> property
>>>>>>> file API and JSON file API.
>>>>>>>
>>>>>>> dt.operator.<operatorname>.classname=<fqcn of operator>
>>>>>>>
>>>>>>> One approach we can take is, if class is of type module, then use
>>>>>>> addModule api
>>>>>>> to add module to the dag, else use addOperator api. The
>>>>>>> initialization
>>>>>>>
>>>>>>> of
>>>>>> object and applying property is generic can be reused by module.
>>>>>>
>>>>>>> Similarly we have Json API where all operators are specified under
>>>>>>> operators array
>>>>>>>
>>>>>>> {
>>>>>>> "operators": [
>>>>>>> {.. operators .. }
>>>>>>> ],
>>>>>>> "streams": [
>>>>>>> {.. streams .. }
>>>>>>> ]
>>>>>>> }
>>>>>>>
>>>>>>> We could add module in the operators array, and based on the object
>>>>>>>
>>>>>>> type
>>>>>> we can
>>>>>>
>>>>>>> consider it as module or operator. The mechanism of initializing
>>>>>>> object
>>>>>>> and applying
>>>>>>> properties remains same.
>>>>>>>
>>>>>>> - Tushar.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>
Re: [APEX-107] Support for specifying module properties on modules.
Posted by Vlad Rozov <v....@datatorrent.com>.
We should either provide Java API that supports interchangeability
between operators and modules or ignore how an operator/module was added
to a DAG.
Thank you,
Vlad
On 11/30/15 20:21, Tushar Gosavi wrote:
> Good point Vlad, if the class implements both Operator and Module
> interface. We will consider it as a module by default in json API, But also
> provide a way to specify his intention whether it should be considered as
> Module or Operator.
>
> If user use a Java API then intention is clear whether the object should be
> considered as Module or Operator (addOperator v/s addModule), will it be
> correct to change the intention of the user internally based on the type of
> component?
>
> - Tushar.
>
>
> On Tue, Dec 1, 2015 at 8:34 AM, Vlad Rozov <v....@datatorrent.com> wrote:
>
>> +1. Modules and operators should be interchangeable. It should be possible
>> for an operator designer to replace it with a module without requirement to
>> recompile all applications that use the existing operator. From application
>> designer view both operators and modules may be considered as a black box
>> with a private implementation that may be monolithic (operator) or
>> distributed (module).
>>
>> Note that a Java class may implement both Operator and Module interface
>> and be added to a DAG using addOperator() method. At run-time (Logical
>> Plan->Physical Plan) Apex platform should handle such classes as modules
>> and may not depend on how a module was added to a DAG, IMO.
>>
>> Thank you,
>>
>> Vlad
>>
>>
>> On 11/30/15 08:41, Tushar Gosavi wrote:
>>
>>> addOperator method in DAG interface takes object of type Operator, unless
>>> we make Module as subclass of Operator we can not use the same API.
>>>
>>> - Tushar.
>>>
>>>
>>> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <th...@datatorrent.com>
>>> wrote:
>>>
>>> +1
>>>> If there are really module specific constructs, they can be added later.
>>>>
>>>> Why is the same not supported in the Java API, i.e. why not allow a
>>>> module
>>>> to be added like an operator and handle the distinction in the
>>>> implementation?
>>>>
>>>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tu...@datatorrent.com>
>>>> wrote:
>>>>
>>>> Hi All,
>>>>> I am going to add a feature to allow use of modules in property file and
>>>>> json api.
>>>>> I am planing to use same "operators" array for adding modules as well as
>>>>> operators. The reasoning behind is that user don't have to worry about
>>>>>
>>>> the
>>>>
>>>>> component being added to the DAG is actually an operator or a module, as
>>>>> the method to specify their properties and to create instance of them is
>>>>> same. And it results in less complex syntax for writing application
>>>>> using
>>>>> json/property file.
>>>>>
>>>>> For example the sample Application specified using Json will look like
>>>>> below.
>>>>> ```json
>>>>> {
>>>>> "operators": [
>>>>> {
>>>>> "name": "operator1",
>>>>> "class": "com.datatorrent.lib.operator.Input",
>>>>> "properties": {
>>>>> "property1": "value1"
>>>>> }
>>>>> },
>>>>> {
>>>>> "name": "module1",
>>>>> "class": "com.datatorrent.module.Module1",
>>>>> "properties": {
>>>>> "property1": "value1"
>>>>> }
>>>>> },
>>>>> ],
>>>>> "streams": [
>>>>> {
>>>>> "name": "s1",
>>>>> "source": {
>>>>> "operatorName": "operator1",
>>>>> "portName": "output"
>>>>> },
>>>>> "sinks": [
>>>>> {
>>>>> "operatorName": "module",
>>>>> "portName": "input"
>>>>> }
>>>>> ]
>>>>> }
>>>>> ]
>>>>> }
>>>>> ```
>>>>>
>>>>> In above example "module1" is of type Module. While processing the Json
>>>>>
>>>> if
>>>>
>>>>> the
>>>>> class of component is of type Module then we will use addModule api to
>>>>>
>>>> the
>>>>
>>>>> DAG.
>>>>>
>>>>> Let me know, what do you think about this.
>>>>>
>>>>> Regards,
>>>>> - Tushar.
>>>>>
>>>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <tu...@datatorrent.com>
>>>>> wrote:
>>>>>
>>>>> A module is superset of operator before expansion in logical plan,
>>>>> except
>>>>> operator
>>>>>> attribute are not applicable to modules. As Tim suggested we will have
>>>>>>
>>>>> to
>>>>> add checks
>>>>>> later to validate these attributes settings, and not allow top level
>>>>>> operator and
>>>>>> module to share same name.
>>>>>>
>>>>>> The same question comes when we want to add module to application using
>>>>>> property
>>>>>> file API and JSON file API.
>>>>>>
>>>>>> dt.operator.<operatorname>.classname=<fqcn of operator>
>>>>>>
>>>>>> One approach we can take is, if class is of type module, then use
>>>>>> addModule api
>>>>>> to add module to the dag, else use addOperator api. The initialization
>>>>>>
>>>>> of
>>>>> object and applying property is generic can be reused by module.
>>>>>> Similarly we have Json API where all operators are specified under
>>>>>> operators array
>>>>>>
>>>>>> {
>>>>>> "operators": [
>>>>>> {.. operators .. }
>>>>>> ],
>>>>>> "streams": [
>>>>>> {.. streams .. }
>>>>>> ]
>>>>>> }
>>>>>>
>>>>>> We could add module in the operators array, and based on the object
>>>>>>
>>>>> type
>>>>> we can
>>>>>> consider it as module or operator. The mechanism of initializing object
>>>>>> and applying
>>>>>> properties remains same.
>>>>>>
>>>>>> - Tushar.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
Re: [APEX-107] Support for specifying module properties on modules.
Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
+1. Modules and operators should be interchangeable.
Should something be recommend in terms of naming convention for Module
Class in that case?
- Chinmay.
~ Chinmay.
On Tue, Dec 1, 2015 at 9:51 AM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
> Good point Vlad, if the class implements both Operator and Module
> interface. We will consider it as a module by default in json API, But also
> provide a way to specify his intention whether it should be considered as
> Module or Operator.
>
> If user use a Java API then intention is clear whether the object should be
> considered as Module or Operator (addOperator v/s addModule), will it be
> correct to change the intention of the user internally based on the type of
> component?
>
> - Tushar.
>
>
> On Tue, Dec 1, 2015 at 8:34 AM, Vlad Rozov <v....@datatorrent.com>
> wrote:
>
> > +1. Modules and operators should be interchangeable. It should be
> possible
> > for an operator designer to replace it with a module without requirement
> to
> > recompile all applications that use the existing operator. From
> application
> > designer view both operators and modules may be considered as a black box
> > with a private implementation that may be monolithic (operator) or
> > distributed (module).
> >
> > Note that a Java class may implement both Operator and Module interface
> > and be added to a DAG using addOperator() method. At run-time (Logical
> > Plan->Physical Plan) Apex platform should handle such classes as modules
> > and may not depend on how a module was added to a DAG, IMO.
> >
> > Thank you,
> >
> > Vlad
> >
> >
> > On 11/30/15 08:41, Tushar Gosavi wrote:
> >
> >> addOperator method in DAG interface takes object of type Operator,
> unless
> >> we make Module as subclass of Operator we can not use the same API.
> >>
> >> - Tushar.
> >>
> >>
> >> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <th...@datatorrent.com>
> >> wrote:
> >>
> >> +1
> >>>
> >>> If there are really module specific constructs, they can be added
> later.
> >>>
> >>> Why is the same not supported in the Java API, i.e. why not allow a
> >>> module
> >>> to be added like an operator and handle the distinction in the
> >>> implementation?
> >>>
> >>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tushar@datatorrent.com
> >
> >>> wrote:
> >>>
> >>> Hi All,
> >>>>
> >>>> I am going to add a feature to allow use of modules in property file
> and
> >>>> json api.
> >>>> I am planing to use same "operators" array for adding modules as well
> as
> >>>> operators. The reasoning behind is that user don't have to worry about
> >>>>
> >>> the
> >>>
> >>>> component being added to the DAG is actually an operator or a module,
> as
> >>>> the method to specify their properties and to create instance of them
> is
> >>>> same. And it results in less complex syntax for writing application
> >>>> using
> >>>> json/property file.
> >>>>
> >>>> For example the sample Application specified using Json will look like
> >>>> below.
> >>>> ```json
> >>>> {
> >>>> "operators": [
> >>>> {
> >>>> "name": "operator1",
> >>>> "class": "com.datatorrent.lib.operator.Input",
> >>>> "properties": {
> >>>> "property1": "value1"
> >>>> }
> >>>> },
> >>>> {
> >>>> "name": "module1",
> >>>> "class": "com.datatorrent.module.Module1",
> >>>> "properties": {
> >>>> "property1": "value1"
> >>>> }
> >>>> },
> >>>> ],
> >>>> "streams": [
> >>>> {
> >>>> "name": "s1",
> >>>> "source": {
> >>>> "operatorName": "operator1",
> >>>> "portName": "output"
> >>>> },
> >>>> "sinks": [
> >>>> {
> >>>> "operatorName": "module",
> >>>> "portName": "input"
> >>>> }
> >>>> ]
> >>>> }
> >>>> ]
> >>>> }
> >>>> ```
> >>>>
> >>>> In above example "module1" is of type Module. While processing the
> Json
> >>>>
> >>> if
> >>>
> >>>> the
> >>>> class of component is of type Module then we will use addModule api to
> >>>>
> >>> the
> >>>
> >>>> DAG.
> >>>>
> >>>> Let me know, what do you think about this.
> >>>>
> >>>> Regards,
> >>>> - Tushar.
> >>>>
> >>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <
> tushar@datatorrent.com>
> >>>> wrote:
> >>>>
> >>>> A module is superset of operator before expansion in logical plan,
> >>>>>
> >>>> except
> >>>
> >>>> operator
> >>>>> attribute are not applicable to modules. As Tim suggested we will
> have
> >>>>>
> >>>> to
> >>>
> >>>> add checks
> >>>>> later to validate these attributes settings, and not allow top level
> >>>>> operator and
> >>>>> module to share same name.
> >>>>>
> >>>>> The same question comes when we want to add module to application
> using
> >>>>> property
> >>>>> file API and JSON file API.
> >>>>>
> >>>>> dt.operator.<operatorname>.classname=<fqcn of operator>
> >>>>>
> >>>>> One approach we can take is, if class is of type module, then use
> >>>>> addModule api
> >>>>> to add module to the dag, else use addOperator api. The
> initialization
> >>>>>
> >>>> of
> >>>
> >>>> object and applying property is generic can be reused by module.
> >>>>>
> >>>>> Similarly we have Json API where all operators are specified under
> >>>>> operators array
> >>>>>
> >>>>> {
> >>>>> "operators": [
> >>>>> {.. operators .. }
> >>>>> ],
> >>>>> "streams": [
> >>>>> {.. streams .. }
> >>>>> ]
> >>>>> }
> >>>>>
> >>>>> We could add module in the operators array, and based on the object
> >>>>>
> >>>> type
> >>>
> >>>> we can
> >>>>> consider it as module or operator. The mechanism of initializing
> object
> >>>>> and applying
> >>>>> properties remains same.
> >>>>>
> >>>>> - Tushar.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >
>
Re: [APEX-107] Support for specifying module properties on modules.
Posted by Tushar Gosavi <tu...@datatorrent.com>.
Good point Vlad, if the class implements both Operator and Module
interface. We will consider it as a module by default in json API, But also
provide a way to specify his intention whether it should be considered as
Module or Operator.
If user use a Java API then intention is clear whether the object should be
considered as Module or Operator (addOperator v/s addModule), will it be
correct to change the intention of the user internally based on the type of
component?
- Tushar.
On Tue, Dec 1, 2015 at 8:34 AM, Vlad Rozov <v....@datatorrent.com> wrote:
> +1. Modules and operators should be interchangeable. It should be possible
> for an operator designer to replace it with a module without requirement to
> recompile all applications that use the existing operator. From application
> designer view both operators and modules may be considered as a black box
> with a private implementation that may be monolithic (operator) or
> distributed (module).
>
> Note that a Java class may implement both Operator and Module interface
> and be added to a DAG using addOperator() method. At run-time (Logical
> Plan->Physical Plan) Apex platform should handle such classes as modules
> and may not depend on how a module was added to a DAG, IMO.
>
> Thank you,
>
> Vlad
>
>
> On 11/30/15 08:41, Tushar Gosavi wrote:
>
>> addOperator method in DAG interface takes object of type Operator, unless
>> we make Module as subclass of Operator we can not use the same API.
>>
>> - Tushar.
>>
>>
>> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <th...@datatorrent.com>
>> wrote:
>>
>> +1
>>>
>>> If there are really module specific constructs, they can be added later.
>>>
>>> Why is the same not supported in the Java API, i.e. why not allow a
>>> module
>>> to be added like an operator and handle the distinction in the
>>> implementation?
>>>
>>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tu...@datatorrent.com>
>>> wrote:
>>>
>>> Hi All,
>>>>
>>>> I am going to add a feature to allow use of modules in property file and
>>>> json api.
>>>> I am planing to use same "operators" array for adding modules as well as
>>>> operators. The reasoning behind is that user don't have to worry about
>>>>
>>> the
>>>
>>>> component being added to the DAG is actually an operator or a module, as
>>>> the method to specify their properties and to create instance of them is
>>>> same. And it results in less complex syntax for writing application
>>>> using
>>>> json/property file.
>>>>
>>>> For example the sample Application specified using Json will look like
>>>> below.
>>>> ```json
>>>> {
>>>> "operators": [
>>>> {
>>>> "name": "operator1",
>>>> "class": "com.datatorrent.lib.operator.Input",
>>>> "properties": {
>>>> "property1": "value1"
>>>> }
>>>> },
>>>> {
>>>> "name": "module1",
>>>> "class": "com.datatorrent.module.Module1",
>>>> "properties": {
>>>> "property1": "value1"
>>>> }
>>>> },
>>>> ],
>>>> "streams": [
>>>> {
>>>> "name": "s1",
>>>> "source": {
>>>> "operatorName": "operator1",
>>>> "portName": "output"
>>>> },
>>>> "sinks": [
>>>> {
>>>> "operatorName": "module",
>>>> "portName": "input"
>>>> }
>>>> ]
>>>> }
>>>> ]
>>>> }
>>>> ```
>>>>
>>>> In above example "module1" is of type Module. While processing the Json
>>>>
>>> if
>>>
>>>> the
>>>> class of component is of type Module then we will use addModule api to
>>>>
>>> the
>>>
>>>> DAG.
>>>>
>>>> Let me know, what do you think about this.
>>>>
>>>> Regards,
>>>> - Tushar.
>>>>
>>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <tu...@datatorrent.com>
>>>> wrote:
>>>>
>>>> A module is superset of operator before expansion in logical plan,
>>>>>
>>>> except
>>>
>>>> operator
>>>>> attribute are not applicable to modules. As Tim suggested we will have
>>>>>
>>>> to
>>>
>>>> add checks
>>>>> later to validate these attributes settings, and not allow top level
>>>>> operator and
>>>>> module to share same name.
>>>>>
>>>>> The same question comes when we want to add module to application using
>>>>> property
>>>>> file API and JSON file API.
>>>>>
>>>>> dt.operator.<operatorname>.classname=<fqcn of operator>
>>>>>
>>>>> One approach we can take is, if class is of type module, then use
>>>>> addModule api
>>>>> to add module to the dag, else use addOperator api. The initialization
>>>>>
>>>> of
>>>
>>>> object and applying property is generic can be reused by module.
>>>>>
>>>>> Similarly we have Json API where all operators are specified under
>>>>> operators array
>>>>>
>>>>> {
>>>>> "operators": [
>>>>> {.. operators .. }
>>>>> ],
>>>>> "streams": [
>>>>> {.. streams .. }
>>>>> ]
>>>>> }
>>>>>
>>>>> We could add module in the operators array, and based on the object
>>>>>
>>>> type
>>>
>>>> we can
>>>>> consider it as module or operator. The mechanism of initializing object
>>>>> and applying
>>>>> properties remains same.
>>>>>
>>>>> - Tushar.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>