You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Chinmay Kolhatkar <ch...@datatorrent.com> on 2016/01/21 18:46:29 UTC

Ability to generate POJO class on the fly

Hi All,

We're planning to add a utility in malhar-library for generating a POJO
class on the fly from given JSON schema.

Use case is where the application is provided with schema and that needs to
be used in one or more operators either as a tuple over the stream OR for
processing.

General Design:
1. Utility will be provided with fqcn of the class and schema definition
provided as json.
2. The schema definition will look like following:
    {
       "fqcn":"<qualified class name>",
       "fields": [
         {
           "name":"field1",
           "type":"long"
         },
         {
           "name":"field2",
           "type": "string"
         }
       ]
    }
3. Supported types identified in "type" JSON field are:
      boolean, char, byte, short, int, float, long, double
4. The output of this utility will be a generated .class file in the form
of FSDataOutputStream.
5. Xbean asm5 library will be used for this.
6. Following methods will be added to the generated class:
    a. Getter/Setter methods for given fields.
    b. simple toString - Generate string equivalent for all the fields
    c. hashCode method - calculate the overall hashCode using individual
field hashcodes, similar to how String.hashCode generates hashCode of
string using chars in it.
    d. equals method - Similar to how String.equals method has done it
using individual char.


Questions I have about the functionality:
1. Should the utility also support nested pojo definition via schema json?
    In such case the field definition can look like following:
    { "name":"nestedField", "type":"<fqcn of nested class>",
"fields":[....] }
    The nested class will be provided

2. Should the utility also support giving out the JarOutputStream which
will have a generated jar?

Please let me know your thoughts.

Thanks,
Chinmay.

Re: Ability to generate POJO class on the fly

Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
Hi All,

I've update the jira with teh summary of this discussion. Jira is
APEXMALHAR-1979.

Also, the pull request is created for this:
https://github.com/apache/incubator-apex-malhar/pull/181

Please review the pull request.

Thanks,
Chinmay.


On Wed, Jan 27, 2016 at 5:14 PM, Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:

> Here is the jira for this: APEXMALHAR-1979
> <https://issues.apache.org/jira/browse/APEXMALHAR-1979>.
>
> If there are no further replies on this topic, I'll update the Jira with
> summary of discussion and create a pull request soon.
>
> Also, If its fine I would skip the nested POJO and Jar output support of
> this utility for now. I'll add the support for it in next iteration if any
> use case comes in.
>
> Thanks,
> Chinmay.
>
>
>
> On Fri, Jan 22, 2016 at 7:16 PM, Chinmay Kolhatkar <
> chinmay@datatorrent.com> wrote:
>
>> OK. We'll do with JSONObject.
>>
>> ~ Chinmay
>> On 22 Jan 2016 19:05, "Bhupesh Chawda" <bh...@datatorrent.com> wrote:
>>
>>> Yes. So in the JDBC output case, a json object could be readily
>>> constructed
>>> by the operator to get a POJO class rather than playing around with JSON
>>> syntax.
>>>
>>> On Fri, Jan 22, 2016 at 6:54 PM, Chinmay Kolhatkar <
>>> chinmay@datatorrent.com>
>>> wrote:
>>>
>>> > @bhupesh, would it help if JSONObject is taken as input to this utility
>>> > rather than json string?
>>> >
>>> > Thanks,
>>> > Chinmay
>>> > On 22 Jan 2016 18:45, "Chinmay Kolhatkar" <ch...@datatorrent.com>
>>> wrote:
>>> >
>>> > > Got it. I think approach 2 is still possible with json based schema.
>>> > > So basically, the operator can create json schema as mention in first
>>> > mail
>>> > > based on column name and type. Then pass that json schema to this
>>> utility
>>> > > to get class to be used directly.
>>> > >
>>> > > Thanks,
>>> > > Chinmay.
>>> > > On 22 Jan 2016 18:38, "Priyanka Gugale" <pr...@datatorrent.com>
>>> > wrote:
>>> > >
>>> > >> Say I need to emit pojo from JDBCInputOperator. I will use on fly
>>> POJO
>>> > >> generator for this purpose instead of providing predefined class.
>>> > >> Now to generate POJO on fly there are two options:
>>> > >> 1. Get json as config from user as you specified.
>>> > >> 2. Based on column name and type generate pojo fields and hence pojo
>>> > class
>>> > >> without much of user configuration. (I am only considering one level
>>> > here,
>>> > >> not the nested pojo)
>>> > >>
>>> > >> Option 1 is must have to provide user a way to control how we
>>> generate
>>> > >> pojo. But option 2 is useful when user simply want to put column
>>> values
>>> > in
>>> > >> some object.
>>> > >>
>>> > >> -Priyanka
>>> > >>
>>> > >> On Fri, Jan 22, 2016 at 6:30 PM, Chinmay Kolhatkar <
>>> > >> chinmay@datatorrent.com>
>>> > >> wrote:
>>> > >>
>>> > >> > Hi Priyanka,
>>> > >> >
>>> > >> > Can you give an example?
>>> > >> >
>>> > >> > Thanks,
>>> > >> > Chinmay.
>>> > >> >
>>> > >> > ~ Chinmay
>>> > >> > On 22 Jan 2016 18:14, "Priyanka Gugale" <priyanka@datatorrent.com
>>> >
>>> > >> wrote:
>>> > >> >
>>> > >> > > We do need configuration in some form like json, but I was
>>> thinking
>>> > we
>>> > >> > can
>>> > >> > > have default wherever possible. This is an option and not
>>> > replacement
>>> > >> to
>>> > >> > > json config.
>>> > >> > >
>>> > >> > > -Priyanka
>>> > >> > >
>>> > >> > > On Fri, Jan 22, 2016 at 4:34 PM, Chinmay Kolhatkar <
>>> > >> > > chinmay@datatorrent.com>
>>> > >> > > wrote:
>>> > >> > >
>>> > >> > > > @priyanka
>>> > >> > > >
>>> > >> > > > Advantage with JSON schema is that one can represent nested
>>> POJO
>>> > >> > > definition
>>> > >> > > > in that. If you're referring to FieldInfo, I'm not sure if
>>> that is
>>> > >> > > possible
>>> > >> > > > there.
>>> > >> > > >
>>> > >> > > > Even if we chose not to have nested POJO now, I think JSON
>>> schema
>>> > >> input
>>> > >> > > to
>>> > >> > > > this utility gives more chance to expand easily later.
>>> > >> > > >
>>> > >> > > >
>>> > >> > > >
>>> > >> > > > On Fri, Jan 22, 2016 at 3:30 PM, Chinmay Kolhatkar <
>>> > >> > > > chinmay@datatorrent.com>
>>> > >> > > > wrote:
>>> > >> > > >
>>> > >> > > > > Sure. We can make this utility to provide byte[] for
>>> compiled
>>> > >> class.
>>> > >> > > > > Though the utility will give a OutputStream object.. One
>>> can get
>>> > >> > byte[]
>>> > >> > > > > out of that.
>>> > >> > > > >
>>> > >> > > > > On Fri, Jan 22, 2016 at 2:56 PM, Timothy Farkas <
>>> > >> tim@datatorrent.com
>>> > >> > >
>>> > >> > > > > wrote:
>>> > >> > > > >
>>> > >> > > > >> Suggestion: +1 for automatic generation of schema. The user
>>> > >> should
>>> > >> > be
>>> > >> > > > able
>>> > >> > > > >> to override the a manually specified schema though.
>>> > >> > > > >>
>>> > >> > > > >> Question: If the mechanism used to generate the class can
>>> > >> produce a
>>> > >> > > > byte[]
>>> > >> > > > >> array, then you can send the byte[] array for the class to
>>> > >> > downstream
>>> > >> > > > >> operators and load the class from the byte[] array in each
>>> > >> operator.
>>> > >> > > An
>>> > >> > > > >> example of how to do this is here:
>>> > >> > > > >>
>>> > >> > > > >>
>>> > >> > > > >>
>>> > >> > > >
>>> > >> > >
>>> > >> >
>>> > >>
>>> >
>>> http://stackoverflow.com/questions/1781091/java-how-to-load-class-stored-as-byte-into-the-jvm
>>> > >> > > > >>
>>> > >> > > > >> Thanks,
>>> > >> > > > >> Tim
>>> > >> > > > >>
>>> > >> > > > >> On Fri, Jan 22, 2016 at 12:59 AM, Priyanka Gugale <
>>> > >> > > > >> priyanka@datatorrent.com>
>>> > >> > > > >> wrote:
>>> > >> > > > >>
>>> > >> > > > >> > Hi,
>>> > >> > > > >> >
>>> > >> > > > >> > Suggestion:
>>> > >> > > > >> > This came up with one of the discussion with Pramod,
>>> will it
>>> > >> be a
>>> > >> > > good
>>> > >> > > > >> > idea, for database input operators to generate pojo
>>> based on
>>> > >> the
>>> > >> > > > >> selected
>>> > >> > > > >> > column field names and types? No need to accept json
>>> input
>>> > from
>>> > >> > > user.
>>> > >> > > > >> >
>>> > >> > > > >> > Question:
>>> > >> > > > >> > How can we share this POJO among multiple operators when
>>> > >> > application
>>> > >> > > > is
>>> > >> > > > >> > already launched and class is generated on the fly?
>>> > >> > > > >> >
>>> > >> > > > >> > -Priyanka
>>> > >> > > > >> >
>>> > >> > > > >> > On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar <
>>> > >> > > > >> > chinmay@datatorrent.com
>>> > >> > > > >> > > wrote:
>>> > >> > > > >> >
>>> > >> > > > >> > > Hi All,
>>> > >> > > > >> > >
>>> > >> > > > >> > > We're planning to add a utility in malhar-library for
>>> > >> > generating a
>>> > >> > > > >> POJO
>>> > >> > > > >> > > class on the fly from given JSON schema.
>>> > >> > > > >> > >
>>> > >> > > > >> > > Use case is where the application is provided with
>>> schema
>>> > and
>>> > >> > that
>>> > >> > > > >> needs
>>> > >> > > > >> > to
>>> > >> > > > >> > > be used in one or more operators either as a tuple
>>> over the
>>> > >> > stream
>>> > >> > > > OR
>>> > >> > > > >> for
>>> > >> > > > >> > > processing.
>>> > >> > > > >> > >
>>> > >> > > > >> > > General Design:
>>> > >> > > > >> > > 1. Utility will be provided with fqcn of the class and
>>> > schema
>>> > >> > > > >> definition
>>> > >> > > > >> > > provided as json.
>>> > >> > > > >> > > 2. The schema definition will look like following:
>>> > >> > > > >> > >     {
>>> > >> > > > >> > >        "fqcn":"<qualified class name>",
>>> > >> > > > >> > >        "fields": [
>>> > >> > > > >> > >          {
>>> > >> > > > >> > >            "name":"field1",
>>> > >> > > > >> > >            "type":"long"
>>> > >> > > > >> > >          },
>>> > >> > > > >> > >          {
>>> > >> > > > >> > >            "name":"field2",
>>> > >> > > > >> > >            "type": "string"
>>> > >> > > > >> > >          }
>>> > >> > > > >> > >        ]
>>> > >> > > > >> > >     }
>>> > >> > > > >> > > 3. Supported types identified in "type" JSON field are:
>>> > >> > > > >> > >       boolean, char, byte, short, int, float, long,
>>> double
>>> > >> > > > >> > > 4. The output of this utility will be a generated
>>> .class
>>> > >> file in
>>> > >> > > the
>>> > >> > > > >> form
>>> > >> > > > >> > > of FSDataOutputStream.
>>> > >> > > > >> > > 5. Xbean asm5 library will be used for this.
>>> > >> > > > >> > > 6. Following methods will be added to the generated
>>> class:
>>> > >> > > > >> > >     a. Getter/Setter methods for given fields.
>>> > >> > > > >> > >     b. simple toString - Generate string equivalent
>>> for all
>>> > >> the
>>> > >> > > > fields
>>> > >> > > > >> > >     c. hashCode method - calculate the overall hashCode
>>> > using
>>> > >> > > > >> individual
>>> > >> > > > >> > > field hashcodes, similar to how String.hashCode
>>> generates
>>> > >> > hashCode
>>> > >> > > > of
>>> > >> > > > >> > > string using chars in it.
>>> > >> > > > >> > >     d. equals method - Similar to how String.equals
>>> method
>>> > >> has
>>> > >> > > done
>>> > >> > > > it
>>> > >> > > > >> > > using individual char.
>>> > >> > > > >> > >
>>> > >> > > > >> > >
>>> > >> > > > >> > > Questions I have about the functionality:
>>> > >> > > > >> > > 1. Should the utility also support nested pojo
>>> definition
>>> > via
>>> > >> > > schema
>>> > >> > > > >> > json?
>>> > >> > > > >> > >     In such case the field definition can look like
>>> > >> following:
>>> > >> > > > >> > >     { "name":"nestedField", "type":"<fqcn of nested
>>> > class>",
>>> > >> > > > >> > > "fields":[....] }
>>> > >> > > > >> > >     The nested class will be provided
>>> > >> > > > >> > >
>>> > >> > > > >> > > 2. Should the utility also support giving out the
>>> > >> > JarOutputStream
>>> > >> > > > >> which
>>> > >> > > > >> > > will have a generated jar?
>>> > >> > > > >> > >
>>> > >> > > > >> > > Please let me know your thoughts.
>>> > >> > > > >> > >
>>> > >> > > > >> > > Thanks,
>>> > >> > > > >> > > Chinmay.
>>> > >> > > > >> > >
>>> > >> > > > >> >
>>> > >> > > > >>
>>> > >> > > > >
>>> > >> > > > >
>>> > >> > > >
>>> > >> > >
>>> > >> >
>>> > >>
>>> > >
>>> >
>>>
>>
>

Re: Ability to generate POJO class on the fly

Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
Here is the jira for this: APEXMALHAR-1979
<https://issues.apache.org/jira/browse/APEXMALHAR-1979>.

If there are no further replies on this topic, I'll update the Jira with
summary of discussion and create a pull request soon.

Also, If its fine I would skip the nested POJO and Jar output support of
this utility for now. I'll add the support for it in next iteration if any
use case comes in.

Thanks,
Chinmay.



On Fri, Jan 22, 2016 at 7:16 PM, Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:

> OK. We'll do with JSONObject.
>
> ~ Chinmay
> On 22 Jan 2016 19:05, "Bhupesh Chawda" <bh...@datatorrent.com> wrote:
>
>> Yes. So in the JDBC output case, a json object could be readily
>> constructed
>> by the operator to get a POJO class rather than playing around with JSON
>> syntax.
>>
>> On Fri, Jan 22, 2016 at 6:54 PM, Chinmay Kolhatkar <
>> chinmay@datatorrent.com>
>> wrote:
>>
>> > @bhupesh, would it help if JSONObject is taken as input to this utility
>> > rather than json string?
>> >
>> > Thanks,
>> > Chinmay
>> > On 22 Jan 2016 18:45, "Chinmay Kolhatkar" <ch...@datatorrent.com>
>> wrote:
>> >
>> > > Got it. I think approach 2 is still possible with json based schema.
>> > > So basically, the operator can create json schema as mention in first
>> > mail
>> > > based on column name and type. Then pass that json schema to this
>> utility
>> > > to get class to be used directly.
>> > >
>> > > Thanks,
>> > > Chinmay.
>> > > On 22 Jan 2016 18:38, "Priyanka Gugale" <pr...@datatorrent.com>
>> > wrote:
>> > >
>> > >> Say I need to emit pojo from JDBCInputOperator. I will use on fly
>> POJO
>> > >> generator for this purpose instead of providing predefined class.
>> > >> Now to generate POJO on fly there are two options:
>> > >> 1. Get json as config from user as you specified.
>> > >> 2. Based on column name and type generate pojo fields and hence pojo
>> > class
>> > >> without much of user configuration. (I am only considering one level
>> > here,
>> > >> not the nested pojo)
>> > >>
>> > >> Option 1 is must have to provide user a way to control how we
>> generate
>> > >> pojo. But option 2 is useful when user simply want to put column
>> values
>> > in
>> > >> some object.
>> > >>
>> > >> -Priyanka
>> > >>
>> > >> On Fri, Jan 22, 2016 at 6:30 PM, Chinmay Kolhatkar <
>> > >> chinmay@datatorrent.com>
>> > >> wrote:
>> > >>
>> > >> > Hi Priyanka,
>> > >> >
>> > >> > Can you give an example?
>> > >> >
>> > >> > Thanks,
>> > >> > Chinmay.
>> > >> >
>> > >> > ~ Chinmay
>> > >> > On 22 Jan 2016 18:14, "Priyanka Gugale" <pr...@datatorrent.com>
>> > >> wrote:
>> > >> >
>> > >> > > We do need configuration in some form like json, but I was
>> thinking
>> > we
>> > >> > can
>> > >> > > have default wherever possible. This is an option and not
>> > replacement
>> > >> to
>> > >> > > json config.
>> > >> > >
>> > >> > > -Priyanka
>> > >> > >
>> > >> > > On Fri, Jan 22, 2016 at 4:34 PM, Chinmay Kolhatkar <
>> > >> > > chinmay@datatorrent.com>
>> > >> > > wrote:
>> > >> > >
>> > >> > > > @priyanka
>> > >> > > >
>> > >> > > > Advantage with JSON schema is that one can represent nested
>> POJO
>> > >> > > definition
>> > >> > > > in that. If you're referring to FieldInfo, I'm not sure if
>> that is
>> > >> > > possible
>> > >> > > > there.
>> > >> > > >
>> > >> > > > Even if we chose not to have nested POJO now, I think JSON
>> schema
>> > >> input
>> > >> > > to
>> > >> > > > this utility gives more chance to expand easily later.
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > > On Fri, Jan 22, 2016 at 3:30 PM, Chinmay Kolhatkar <
>> > >> > > > chinmay@datatorrent.com>
>> > >> > > > wrote:
>> > >> > > >
>> > >> > > > > Sure. We can make this utility to provide byte[] for compiled
>> > >> class.
>> > >> > > > > Though the utility will give a OutputStream object.. One can
>> get
>> > >> > byte[]
>> > >> > > > > out of that.
>> > >> > > > >
>> > >> > > > > On Fri, Jan 22, 2016 at 2:56 PM, Timothy Farkas <
>> > >> tim@datatorrent.com
>> > >> > >
>> > >> > > > > wrote:
>> > >> > > > >
>> > >> > > > >> Suggestion: +1 for automatic generation of schema. The user
>> > >> should
>> > >> > be
>> > >> > > > able
>> > >> > > > >> to override the a manually specified schema though.
>> > >> > > > >>
>> > >> > > > >> Question: If the mechanism used to generate the class can
>> > >> produce a
>> > >> > > > byte[]
>> > >> > > > >> array, then you can send the byte[] array for the class to
>> > >> > downstream
>> > >> > > > >> operators and load the class from the byte[] array in each
>> > >> operator.
>> > >> > > An
>> > >> > > > >> example of how to do this is here:
>> > >> > > > >>
>> > >> > > > >>
>> > >> > > > >>
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>> http://stackoverflow.com/questions/1781091/java-how-to-load-class-stored-as-byte-into-the-jvm
>> > >> > > > >>
>> > >> > > > >> Thanks,
>> > >> > > > >> Tim
>> > >> > > > >>
>> > >> > > > >> On Fri, Jan 22, 2016 at 12:59 AM, Priyanka Gugale <
>> > >> > > > >> priyanka@datatorrent.com>
>> > >> > > > >> wrote:
>> > >> > > > >>
>> > >> > > > >> > Hi,
>> > >> > > > >> >
>> > >> > > > >> > Suggestion:
>> > >> > > > >> > This came up with one of the discussion with Pramod, will
>> it
>> > >> be a
>> > >> > > good
>> > >> > > > >> > idea, for database input operators to generate pojo based
>> on
>> > >> the
>> > >> > > > >> selected
>> > >> > > > >> > column field names and types? No need to accept json input
>> > from
>> > >> > > user.
>> > >> > > > >> >
>> > >> > > > >> > Question:
>> > >> > > > >> > How can we share this POJO among multiple operators when
>> > >> > application
>> > >> > > > is
>> > >> > > > >> > already launched and class is generated on the fly?
>> > >> > > > >> >
>> > >> > > > >> > -Priyanka
>> > >> > > > >> >
>> > >> > > > >> > On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar <
>> > >> > > > >> > chinmay@datatorrent.com
>> > >> > > > >> > > wrote:
>> > >> > > > >> >
>> > >> > > > >> > > Hi All,
>> > >> > > > >> > >
>> > >> > > > >> > > We're planning to add a utility in malhar-library for
>> > >> > generating a
>> > >> > > > >> POJO
>> > >> > > > >> > > class on the fly from given JSON schema.
>> > >> > > > >> > >
>> > >> > > > >> > > Use case is where the application is provided with
>> schema
>> > and
>> > >> > that
>> > >> > > > >> needs
>> > >> > > > >> > to
>> > >> > > > >> > > be used in one or more operators either as a tuple over
>> the
>> > >> > stream
>> > >> > > > OR
>> > >> > > > >> for
>> > >> > > > >> > > processing.
>> > >> > > > >> > >
>> > >> > > > >> > > General Design:
>> > >> > > > >> > > 1. Utility will be provided with fqcn of the class and
>> > schema
>> > >> > > > >> definition
>> > >> > > > >> > > provided as json.
>> > >> > > > >> > > 2. The schema definition will look like following:
>> > >> > > > >> > >     {
>> > >> > > > >> > >        "fqcn":"<qualified class name>",
>> > >> > > > >> > >        "fields": [
>> > >> > > > >> > >          {
>> > >> > > > >> > >            "name":"field1",
>> > >> > > > >> > >            "type":"long"
>> > >> > > > >> > >          },
>> > >> > > > >> > >          {
>> > >> > > > >> > >            "name":"field2",
>> > >> > > > >> > >            "type": "string"
>> > >> > > > >> > >          }
>> > >> > > > >> > >        ]
>> > >> > > > >> > >     }
>> > >> > > > >> > > 3. Supported types identified in "type" JSON field are:
>> > >> > > > >> > >       boolean, char, byte, short, int, float, long,
>> double
>> > >> > > > >> > > 4. The output of this utility will be a generated .class
>> > >> file in
>> > >> > > the
>> > >> > > > >> form
>> > >> > > > >> > > of FSDataOutputStream.
>> > >> > > > >> > > 5. Xbean asm5 library will be used for this.
>> > >> > > > >> > > 6. Following methods will be added to the generated
>> class:
>> > >> > > > >> > >     a. Getter/Setter methods for given fields.
>> > >> > > > >> > >     b. simple toString - Generate string equivalent for
>> all
>> > >> the
>> > >> > > > fields
>> > >> > > > >> > >     c. hashCode method - calculate the overall hashCode
>> > using
>> > >> > > > >> individual
>> > >> > > > >> > > field hashcodes, similar to how String.hashCode
>> generates
>> > >> > hashCode
>> > >> > > > of
>> > >> > > > >> > > string using chars in it.
>> > >> > > > >> > >     d. equals method - Similar to how String.equals
>> method
>> > >> has
>> > >> > > done
>> > >> > > > it
>> > >> > > > >> > > using individual char.
>> > >> > > > >> > >
>> > >> > > > >> > >
>> > >> > > > >> > > Questions I have about the functionality:
>> > >> > > > >> > > 1. Should the utility also support nested pojo
>> definition
>> > via
>> > >> > > schema
>> > >> > > > >> > json?
>> > >> > > > >> > >     In such case the field definition can look like
>> > >> following:
>> > >> > > > >> > >     { "name":"nestedField", "type":"<fqcn of nested
>> > class>",
>> > >> > > > >> > > "fields":[....] }
>> > >> > > > >> > >     The nested class will be provided
>> > >> > > > >> > >
>> > >> > > > >> > > 2. Should the utility also support giving out the
>> > >> > JarOutputStream
>> > >> > > > >> which
>> > >> > > > >> > > will have a generated jar?
>> > >> > > > >> > >
>> > >> > > > >> > > Please let me know your thoughts.
>> > >> > > > >> > >
>> > >> > > > >> > > Thanks,
>> > >> > > > >> > > Chinmay.
>> > >> > > > >> > >
>> > >> > > > >> >
>> > >> > > > >>
>> > >> > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>>
>

Re: Ability to generate POJO class on the fly

Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
OK. We'll do with JSONObject.

~ Chinmay
On 22 Jan 2016 19:05, "Bhupesh Chawda" <bh...@datatorrent.com> wrote:

> Yes. So in the JDBC output case, a json object could be readily constructed
> by the operator to get a POJO class rather than playing around with JSON
> syntax.
>
> On Fri, Jan 22, 2016 at 6:54 PM, Chinmay Kolhatkar <
> chinmay@datatorrent.com>
> wrote:
>
> > @bhupesh, would it help if JSONObject is taken as input to this utility
> > rather than json string?
> >
> > Thanks,
> > Chinmay
> > On 22 Jan 2016 18:45, "Chinmay Kolhatkar" <ch...@datatorrent.com>
> wrote:
> >
> > > Got it. I think approach 2 is still possible with json based schema.
> > > So basically, the operator can create json schema as mention in first
> > mail
> > > based on column name and type. Then pass that json schema to this
> utility
> > > to get class to be used directly.
> > >
> > > Thanks,
> > > Chinmay.
> > > On 22 Jan 2016 18:38, "Priyanka Gugale" <pr...@datatorrent.com>
> > wrote:
> > >
> > >> Say I need to emit pojo from JDBCInputOperator. I will use on fly POJO
> > >> generator for this purpose instead of providing predefined class.
> > >> Now to generate POJO on fly there are two options:
> > >> 1. Get json as config from user as you specified.
> > >> 2. Based on column name and type generate pojo fields and hence pojo
> > class
> > >> without much of user configuration. (I am only considering one level
> > here,
> > >> not the nested pojo)
> > >>
> > >> Option 1 is must have to provide user a way to control how we generate
> > >> pojo. But option 2 is useful when user simply want to put column
> values
> > in
> > >> some object.
> > >>
> > >> -Priyanka
> > >>
> > >> On Fri, Jan 22, 2016 at 6:30 PM, Chinmay Kolhatkar <
> > >> chinmay@datatorrent.com>
> > >> wrote:
> > >>
> > >> > Hi Priyanka,
> > >> >
> > >> > Can you give an example?
> > >> >
> > >> > Thanks,
> > >> > Chinmay.
> > >> >
> > >> > ~ Chinmay
> > >> > On 22 Jan 2016 18:14, "Priyanka Gugale" <pr...@datatorrent.com>
> > >> wrote:
> > >> >
> > >> > > We do need configuration in some form like json, but I was
> thinking
> > we
> > >> > can
> > >> > > have default wherever possible. This is an option and not
> > replacement
> > >> to
> > >> > > json config.
> > >> > >
> > >> > > -Priyanka
> > >> > >
> > >> > > On Fri, Jan 22, 2016 at 4:34 PM, Chinmay Kolhatkar <
> > >> > > chinmay@datatorrent.com>
> > >> > > wrote:
> > >> > >
> > >> > > > @priyanka
> > >> > > >
> > >> > > > Advantage with JSON schema is that one can represent nested POJO
> > >> > > definition
> > >> > > > in that. If you're referring to FieldInfo, I'm not sure if that
> is
> > >> > > possible
> > >> > > > there.
> > >> > > >
> > >> > > > Even if we chose not to have nested POJO now, I think JSON
> schema
> > >> input
> > >> > > to
> > >> > > > this utility gives more chance to expand easily later.
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Fri, Jan 22, 2016 at 3:30 PM, Chinmay Kolhatkar <
> > >> > > > chinmay@datatorrent.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Sure. We can make this utility to provide byte[] for compiled
> > >> class.
> > >> > > > > Though the utility will give a OutputStream object.. One can
> get
> > >> > byte[]
> > >> > > > > out of that.
> > >> > > > >
> > >> > > > > On Fri, Jan 22, 2016 at 2:56 PM, Timothy Farkas <
> > >> tim@datatorrent.com
> > >> > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > >> Suggestion: +1 for automatic generation of schema. The user
> > >> should
> > >> > be
> > >> > > > able
> > >> > > > >> to override the a manually specified schema though.
> > >> > > > >>
> > >> > > > >> Question: If the mechanism used to generate the class can
> > >> produce a
> > >> > > > byte[]
> > >> > > > >> array, then you can send the byte[] array for the class to
> > >> > downstream
> > >> > > > >> operators and load the class from the byte[] array in each
> > >> operator.
> > >> > > An
> > >> > > > >> example of how to do this is here:
> > >> > > > >>
> > >> > > > >>
> > >> > > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://stackoverflow.com/questions/1781091/java-how-to-load-class-stored-as-byte-into-the-jvm
> > >> > > > >>
> > >> > > > >> Thanks,
> > >> > > > >> Tim
> > >> > > > >>
> > >> > > > >> On Fri, Jan 22, 2016 at 12:59 AM, Priyanka Gugale <
> > >> > > > >> priyanka@datatorrent.com>
> > >> > > > >> wrote:
> > >> > > > >>
> > >> > > > >> > Hi,
> > >> > > > >> >
> > >> > > > >> > Suggestion:
> > >> > > > >> > This came up with one of the discussion with Pramod, will
> it
> > >> be a
> > >> > > good
> > >> > > > >> > idea, for database input operators to generate pojo based
> on
> > >> the
> > >> > > > >> selected
> > >> > > > >> > column field names and types? No need to accept json input
> > from
> > >> > > user.
> > >> > > > >> >
> > >> > > > >> > Question:
> > >> > > > >> > How can we share this POJO among multiple operators when
> > >> > application
> > >> > > > is
> > >> > > > >> > already launched and class is generated on the fly?
> > >> > > > >> >
> > >> > > > >> > -Priyanka
> > >> > > > >> >
> > >> > > > >> > On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar <
> > >> > > > >> > chinmay@datatorrent.com
> > >> > > > >> > > wrote:
> > >> > > > >> >
> > >> > > > >> > > Hi All,
> > >> > > > >> > >
> > >> > > > >> > > We're planning to add a utility in malhar-library for
> > >> > generating a
> > >> > > > >> POJO
> > >> > > > >> > > class on the fly from given JSON schema.
> > >> > > > >> > >
> > >> > > > >> > > Use case is where the application is provided with schema
> > and
> > >> > that
> > >> > > > >> needs
> > >> > > > >> > to
> > >> > > > >> > > be used in one or more operators either as a tuple over
> the
> > >> > stream
> > >> > > > OR
> > >> > > > >> for
> > >> > > > >> > > processing.
> > >> > > > >> > >
> > >> > > > >> > > General Design:
> > >> > > > >> > > 1. Utility will be provided with fqcn of the class and
> > schema
> > >> > > > >> definition
> > >> > > > >> > > provided as json.
> > >> > > > >> > > 2. The schema definition will look like following:
> > >> > > > >> > >     {
> > >> > > > >> > >        "fqcn":"<qualified class name>",
> > >> > > > >> > >        "fields": [
> > >> > > > >> > >          {
> > >> > > > >> > >            "name":"field1",
> > >> > > > >> > >            "type":"long"
> > >> > > > >> > >          },
> > >> > > > >> > >          {
> > >> > > > >> > >            "name":"field2",
> > >> > > > >> > >            "type": "string"
> > >> > > > >> > >          }
> > >> > > > >> > >        ]
> > >> > > > >> > >     }
> > >> > > > >> > > 3. Supported types identified in "type" JSON field are:
> > >> > > > >> > >       boolean, char, byte, short, int, float, long,
> double
> > >> > > > >> > > 4. The output of this utility will be a generated .class
> > >> file in
> > >> > > the
> > >> > > > >> form
> > >> > > > >> > > of FSDataOutputStream.
> > >> > > > >> > > 5. Xbean asm5 library will be used for this.
> > >> > > > >> > > 6. Following methods will be added to the generated
> class:
> > >> > > > >> > >     a. Getter/Setter methods for given fields.
> > >> > > > >> > >     b. simple toString - Generate string equivalent for
> all
> > >> the
> > >> > > > fields
> > >> > > > >> > >     c. hashCode method - calculate the overall hashCode
> > using
> > >> > > > >> individual
> > >> > > > >> > > field hashcodes, similar to how String.hashCode generates
> > >> > hashCode
> > >> > > > of
> > >> > > > >> > > string using chars in it.
> > >> > > > >> > >     d. equals method - Similar to how String.equals
> method
> > >> has
> > >> > > done
> > >> > > > it
> > >> > > > >> > > using individual char.
> > >> > > > >> > >
> > >> > > > >> > >
> > >> > > > >> > > Questions I have about the functionality:
> > >> > > > >> > > 1. Should the utility also support nested pojo definition
> > via
> > >> > > schema
> > >> > > > >> > json?
> > >> > > > >> > >     In such case the field definition can look like
> > >> following:
> > >> > > > >> > >     { "name":"nestedField", "type":"<fqcn of nested
> > class>",
> > >> > > > >> > > "fields":[....] }
> > >> > > > >> > >     The nested class will be provided
> > >> > > > >> > >
> > >> > > > >> > > 2. Should the utility also support giving out the
> > >> > JarOutputStream
> > >> > > > >> which
> > >> > > > >> > > will have a generated jar?
> > >> > > > >> > >
> > >> > > > >> > > Please let me know your thoughts.
> > >> > > > >> > >
> > >> > > > >> > > Thanks,
> > >> > > > >> > > Chinmay.
> > >> > > > >> > >
> > >> > > > >> >
> > >> > > > >>
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: Ability to generate POJO class on the fly

Posted by Bhupesh Chawda <bh...@datatorrent.com>.
Yes. So in the JDBC output case, a json object could be readily constructed
by the operator to get a POJO class rather than playing around with JSON
syntax.

On Fri, Jan 22, 2016 at 6:54 PM, Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:

> @bhupesh, would it help if JSONObject is taken as input to this utility
> rather than json string?
>
> Thanks,
> Chinmay
> On 22 Jan 2016 18:45, "Chinmay Kolhatkar" <ch...@datatorrent.com> wrote:
>
> > Got it. I think approach 2 is still possible with json based schema.
> > So basically, the operator can create json schema as mention in first
> mail
> > based on column name and type. Then pass that json schema to this utility
> > to get class to be used directly.
> >
> > Thanks,
> > Chinmay.
> > On 22 Jan 2016 18:38, "Priyanka Gugale" <pr...@datatorrent.com>
> wrote:
> >
> >> Say I need to emit pojo from JDBCInputOperator. I will use on fly POJO
> >> generator for this purpose instead of providing predefined class.
> >> Now to generate POJO on fly there are two options:
> >> 1. Get json as config from user as you specified.
> >> 2. Based on column name and type generate pojo fields and hence pojo
> class
> >> without much of user configuration. (I am only considering one level
> here,
> >> not the nested pojo)
> >>
> >> Option 1 is must have to provide user a way to control how we generate
> >> pojo. But option 2 is useful when user simply want to put column values
> in
> >> some object.
> >>
> >> -Priyanka
> >>
> >> On Fri, Jan 22, 2016 at 6:30 PM, Chinmay Kolhatkar <
> >> chinmay@datatorrent.com>
> >> wrote:
> >>
> >> > Hi Priyanka,
> >> >
> >> > Can you give an example?
> >> >
> >> > Thanks,
> >> > Chinmay.
> >> >
> >> > ~ Chinmay
> >> > On 22 Jan 2016 18:14, "Priyanka Gugale" <pr...@datatorrent.com>
> >> wrote:
> >> >
> >> > > We do need configuration in some form like json, but I was thinking
> we
> >> > can
> >> > > have default wherever possible. This is an option and not
> replacement
> >> to
> >> > > json config.
> >> > >
> >> > > -Priyanka
> >> > >
> >> > > On Fri, Jan 22, 2016 at 4:34 PM, Chinmay Kolhatkar <
> >> > > chinmay@datatorrent.com>
> >> > > wrote:
> >> > >
> >> > > > @priyanka
> >> > > >
> >> > > > Advantage with JSON schema is that one can represent nested POJO
> >> > > definition
> >> > > > in that. If you're referring to FieldInfo, I'm not sure if that is
> >> > > possible
> >> > > > there.
> >> > > >
> >> > > > Even if we chose not to have nested POJO now, I think JSON schema
> >> input
> >> > > to
> >> > > > this utility gives more chance to expand easily later.
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Fri, Jan 22, 2016 at 3:30 PM, Chinmay Kolhatkar <
> >> > > > chinmay@datatorrent.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Sure. We can make this utility to provide byte[] for compiled
> >> class.
> >> > > > > Though the utility will give a OutputStream object.. One can get
> >> > byte[]
> >> > > > > out of that.
> >> > > > >
> >> > > > > On Fri, Jan 22, 2016 at 2:56 PM, Timothy Farkas <
> >> tim@datatorrent.com
> >> > >
> >> > > > > wrote:
> >> > > > >
> >> > > > >> Suggestion: +1 for automatic generation of schema. The user
> >> should
> >> > be
> >> > > > able
> >> > > > >> to override the a manually specified schema though.
> >> > > > >>
> >> > > > >> Question: If the mechanism used to generate the class can
> >> produce a
> >> > > > byte[]
> >> > > > >> array, then you can send the byte[] array for the class to
> >> > downstream
> >> > > > >> operators and load the class from the byte[] array in each
> >> operator.
> >> > > An
> >> > > > >> example of how to do this is here:
> >> > > > >>
> >> > > > >>
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
> http://stackoverflow.com/questions/1781091/java-how-to-load-class-stored-as-byte-into-the-jvm
> >> > > > >>
> >> > > > >> Thanks,
> >> > > > >> Tim
> >> > > > >>
> >> > > > >> On Fri, Jan 22, 2016 at 12:59 AM, Priyanka Gugale <
> >> > > > >> priyanka@datatorrent.com>
> >> > > > >> wrote:
> >> > > > >>
> >> > > > >> > Hi,
> >> > > > >> >
> >> > > > >> > Suggestion:
> >> > > > >> > This came up with one of the discussion with Pramod, will it
> >> be a
> >> > > good
> >> > > > >> > idea, for database input operators to generate pojo based on
> >> the
> >> > > > >> selected
> >> > > > >> > column field names and types? No need to accept json input
> from
> >> > > user.
> >> > > > >> >
> >> > > > >> > Question:
> >> > > > >> > How can we share this POJO among multiple operators when
> >> > application
> >> > > > is
> >> > > > >> > already launched and class is generated on the fly?
> >> > > > >> >
> >> > > > >> > -Priyanka
> >> > > > >> >
> >> > > > >> > On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar <
> >> > > > >> > chinmay@datatorrent.com
> >> > > > >> > > wrote:
> >> > > > >> >
> >> > > > >> > > Hi All,
> >> > > > >> > >
> >> > > > >> > > We're planning to add a utility in malhar-library for
> >> > generating a
> >> > > > >> POJO
> >> > > > >> > > class on the fly from given JSON schema.
> >> > > > >> > >
> >> > > > >> > > Use case is where the application is provided with schema
> and
> >> > that
> >> > > > >> needs
> >> > > > >> > to
> >> > > > >> > > be used in one or more operators either as a tuple over the
> >> > stream
> >> > > > OR
> >> > > > >> for
> >> > > > >> > > processing.
> >> > > > >> > >
> >> > > > >> > > General Design:
> >> > > > >> > > 1. Utility will be provided with fqcn of the class and
> schema
> >> > > > >> definition
> >> > > > >> > > provided as json.
> >> > > > >> > > 2. The schema definition will look like following:
> >> > > > >> > >     {
> >> > > > >> > >        "fqcn":"<qualified class name>",
> >> > > > >> > >        "fields": [
> >> > > > >> > >          {
> >> > > > >> > >            "name":"field1",
> >> > > > >> > >            "type":"long"
> >> > > > >> > >          },
> >> > > > >> > >          {
> >> > > > >> > >            "name":"field2",
> >> > > > >> > >            "type": "string"
> >> > > > >> > >          }
> >> > > > >> > >        ]
> >> > > > >> > >     }
> >> > > > >> > > 3. Supported types identified in "type" JSON field are:
> >> > > > >> > >       boolean, char, byte, short, int, float, long, double
> >> > > > >> > > 4. The output of this utility will be a generated .class
> >> file in
> >> > > the
> >> > > > >> form
> >> > > > >> > > of FSDataOutputStream.
> >> > > > >> > > 5. Xbean asm5 library will be used for this.
> >> > > > >> > > 6. Following methods will be added to the generated class:
> >> > > > >> > >     a. Getter/Setter methods for given fields.
> >> > > > >> > >     b. simple toString - Generate string equivalent for all
> >> the
> >> > > > fields
> >> > > > >> > >     c. hashCode method - calculate the overall hashCode
> using
> >> > > > >> individual
> >> > > > >> > > field hashcodes, similar to how String.hashCode generates
> >> > hashCode
> >> > > > of
> >> > > > >> > > string using chars in it.
> >> > > > >> > >     d. equals method - Similar to how String.equals method
> >> has
> >> > > done
> >> > > > it
> >> > > > >> > > using individual char.
> >> > > > >> > >
> >> > > > >> > >
> >> > > > >> > > Questions I have about the functionality:
> >> > > > >> > > 1. Should the utility also support nested pojo definition
> via
> >> > > schema
> >> > > > >> > json?
> >> > > > >> > >     In such case the field definition can look like
> >> following:
> >> > > > >> > >     { "name":"nestedField", "type":"<fqcn of nested
> class>",
> >> > > > >> > > "fields":[....] }
> >> > > > >> > >     The nested class will be provided
> >> > > > >> > >
> >> > > > >> > > 2. Should the utility also support giving out the
> >> > JarOutputStream
> >> > > > >> which
> >> > > > >> > > will have a generated jar?
> >> > > > >> > >
> >> > > > >> > > Please let me know your thoughts.
> >> > > > >> > >
> >> > > > >> > > Thanks,
> >> > > > >> > > Chinmay.
> >> > > > >> > >
> >> > > > >> >
> >> > > > >>
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: Ability to generate POJO class on the fly

Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
@bhupesh, would it help if JSONObject is taken as input to this utility
rather than json string?

Thanks,
Chinmay
On 22 Jan 2016 18:45, "Chinmay Kolhatkar" <ch...@datatorrent.com> wrote:

> Got it. I think approach 2 is still possible with json based schema.
> So basically, the operator can create json schema as mention in first mail
> based on column name and type. Then pass that json schema to this utility
> to get class to be used directly.
>
> Thanks,
> Chinmay.
> On 22 Jan 2016 18:38, "Priyanka Gugale" <pr...@datatorrent.com> wrote:
>
>> Say I need to emit pojo from JDBCInputOperator. I will use on fly POJO
>> generator for this purpose instead of providing predefined class.
>> Now to generate POJO on fly there are two options:
>> 1. Get json as config from user as you specified.
>> 2. Based on column name and type generate pojo fields and hence pojo class
>> without much of user configuration. (I am only considering one level here,
>> not the nested pojo)
>>
>> Option 1 is must have to provide user a way to control how we generate
>> pojo. But option 2 is useful when user simply want to put column values in
>> some object.
>>
>> -Priyanka
>>
>> On Fri, Jan 22, 2016 at 6:30 PM, Chinmay Kolhatkar <
>> chinmay@datatorrent.com>
>> wrote:
>>
>> > Hi Priyanka,
>> >
>> > Can you give an example?
>> >
>> > Thanks,
>> > Chinmay.
>> >
>> > ~ Chinmay
>> > On 22 Jan 2016 18:14, "Priyanka Gugale" <pr...@datatorrent.com>
>> wrote:
>> >
>> > > We do need configuration in some form like json, but I was thinking we
>> > can
>> > > have default wherever possible. This is an option and not replacement
>> to
>> > > json config.
>> > >
>> > > -Priyanka
>> > >
>> > > On Fri, Jan 22, 2016 at 4:34 PM, Chinmay Kolhatkar <
>> > > chinmay@datatorrent.com>
>> > > wrote:
>> > >
>> > > > @priyanka
>> > > >
>> > > > Advantage with JSON schema is that one can represent nested POJO
>> > > definition
>> > > > in that. If you're referring to FieldInfo, I'm not sure if that is
>> > > possible
>> > > > there.
>> > > >
>> > > > Even if we chose not to have nested POJO now, I think JSON schema
>> input
>> > > to
>> > > > this utility gives more chance to expand easily later.
>> > > >
>> > > >
>> > > >
>> > > > On Fri, Jan 22, 2016 at 3:30 PM, Chinmay Kolhatkar <
>> > > > chinmay@datatorrent.com>
>> > > > wrote:
>> > > >
>> > > > > Sure. We can make this utility to provide byte[] for compiled
>> class.
>> > > > > Though the utility will give a OutputStream object.. One can get
>> > byte[]
>> > > > > out of that.
>> > > > >
>> > > > > On Fri, Jan 22, 2016 at 2:56 PM, Timothy Farkas <
>> tim@datatorrent.com
>> > >
>> > > > > wrote:
>> > > > >
>> > > > >> Suggestion: +1 for automatic generation of schema. The user
>> should
>> > be
>> > > > able
>> > > > >> to override the a manually specified schema though.
>> > > > >>
>> > > > >> Question: If the mechanism used to generate the class can
>> produce a
>> > > > byte[]
>> > > > >> array, then you can send the byte[] array for the class to
>> > downstream
>> > > > >> operators and load the class from the byte[] array in each
>> operator.
>> > > An
>> > > > >> example of how to do this is here:
>> > > > >>
>> > > > >>
>> > > > >>
>> > > >
>> > >
>> >
>> http://stackoverflow.com/questions/1781091/java-how-to-load-class-stored-as-byte-into-the-jvm
>> > > > >>
>> > > > >> Thanks,
>> > > > >> Tim
>> > > > >>
>> > > > >> On Fri, Jan 22, 2016 at 12:59 AM, Priyanka Gugale <
>> > > > >> priyanka@datatorrent.com>
>> > > > >> wrote:
>> > > > >>
>> > > > >> > Hi,
>> > > > >> >
>> > > > >> > Suggestion:
>> > > > >> > This came up with one of the discussion with Pramod, will it
>> be a
>> > > good
>> > > > >> > idea, for database input operators to generate pojo based on
>> the
>> > > > >> selected
>> > > > >> > column field names and types? No need to accept json input from
>> > > user.
>> > > > >> >
>> > > > >> > Question:
>> > > > >> > How can we share this POJO among multiple operators when
>> > application
>> > > > is
>> > > > >> > already launched and class is generated on the fly?
>> > > > >> >
>> > > > >> > -Priyanka
>> > > > >> >
>> > > > >> > On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar <
>> > > > >> > chinmay@datatorrent.com
>> > > > >> > > wrote:
>> > > > >> >
>> > > > >> > > Hi All,
>> > > > >> > >
>> > > > >> > > We're planning to add a utility in malhar-library for
>> > generating a
>> > > > >> POJO
>> > > > >> > > class on the fly from given JSON schema.
>> > > > >> > >
>> > > > >> > > Use case is where the application is provided with schema and
>> > that
>> > > > >> needs
>> > > > >> > to
>> > > > >> > > be used in one or more operators either as a tuple over the
>> > stream
>> > > > OR
>> > > > >> for
>> > > > >> > > processing.
>> > > > >> > >
>> > > > >> > > General Design:
>> > > > >> > > 1. Utility will be provided with fqcn of the class and schema
>> > > > >> definition
>> > > > >> > > provided as json.
>> > > > >> > > 2. The schema definition will look like following:
>> > > > >> > >     {
>> > > > >> > >        "fqcn":"<qualified class name>",
>> > > > >> > >        "fields": [
>> > > > >> > >          {
>> > > > >> > >            "name":"field1",
>> > > > >> > >            "type":"long"
>> > > > >> > >          },
>> > > > >> > >          {
>> > > > >> > >            "name":"field2",
>> > > > >> > >            "type": "string"
>> > > > >> > >          }
>> > > > >> > >        ]
>> > > > >> > >     }
>> > > > >> > > 3. Supported types identified in "type" JSON field are:
>> > > > >> > >       boolean, char, byte, short, int, float, long, double
>> > > > >> > > 4. The output of this utility will be a generated .class
>> file in
>> > > the
>> > > > >> form
>> > > > >> > > of FSDataOutputStream.
>> > > > >> > > 5. Xbean asm5 library will be used for this.
>> > > > >> > > 6. Following methods will be added to the generated class:
>> > > > >> > >     a. Getter/Setter methods for given fields.
>> > > > >> > >     b. simple toString - Generate string equivalent for all
>> the
>> > > > fields
>> > > > >> > >     c. hashCode method - calculate the overall hashCode using
>> > > > >> individual
>> > > > >> > > field hashcodes, similar to how String.hashCode generates
>> > hashCode
>> > > > of
>> > > > >> > > string using chars in it.
>> > > > >> > >     d. equals method - Similar to how String.equals method
>> has
>> > > done
>> > > > it
>> > > > >> > > using individual char.
>> > > > >> > >
>> > > > >> > >
>> > > > >> > > Questions I have about the functionality:
>> > > > >> > > 1. Should the utility also support nested pojo definition via
>> > > schema
>> > > > >> > json?
>> > > > >> > >     In such case the field definition can look like
>> following:
>> > > > >> > >     { "name":"nestedField", "type":"<fqcn of nested class>",
>> > > > >> > > "fields":[....] }
>> > > > >> > >     The nested class will be provided
>> > > > >> > >
>> > > > >> > > 2. Should the utility also support giving out the
>> > JarOutputStream
>> > > > >> which
>> > > > >> > > will have a generated jar?
>> > > > >> > >
>> > > > >> > > Please let me know your thoughts.
>> > > > >> > >
>> > > > >> > > Thanks,
>> > > > >> > > Chinmay.
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: Ability to generate POJO class on the fly

Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
Got it. I think approach 2 is still possible with json based schema.
So basically, the operator can create json schema as mention in first mail
based on column name and type. Then pass that json schema to this utility
to get class to be used directly.

Thanks,
Chinmay.
On 22 Jan 2016 18:38, "Priyanka Gugale" <pr...@datatorrent.com> wrote:

> Say I need to emit pojo from JDBCInputOperator. I will use on fly POJO
> generator for this purpose instead of providing predefined class.
> Now to generate POJO on fly there are two options:
> 1. Get json as config from user as you specified.
> 2. Based on column name and type generate pojo fields and hence pojo class
> without much of user configuration. (I am only considering one level here,
> not the nested pojo)
>
> Option 1 is must have to provide user a way to control how we generate
> pojo. But option 2 is useful when user simply want to put column values in
> some object.
>
> -Priyanka
>
> On Fri, Jan 22, 2016 at 6:30 PM, Chinmay Kolhatkar <
> chinmay@datatorrent.com>
> wrote:
>
> > Hi Priyanka,
> >
> > Can you give an example?
> >
> > Thanks,
> > Chinmay.
> >
> > ~ Chinmay
> > On 22 Jan 2016 18:14, "Priyanka Gugale" <pr...@datatorrent.com>
> wrote:
> >
> > > We do need configuration in some form like json, but I was thinking we
> > can
> > > have default wherever possible. This is an option and not replacement
> to
> > > json config.
> > >
> > > -Priyanka
> > >
> > > On Fri, Jan 22, 2016 at 4:34 PM, Chinmay Kolhatkar <
> > > chinmay@datatorrent.com>
> > > wrote:
> > >
> > > > @priyanka
> > > >
> > > > Advantage with JSON schema is that one can represent nested POJO
> > > definition
> > > > in that. If you're referring to FieldInfo, I'm not sure if that is
> > > possible
> > > > there.
> > > >
> > > > Even if we chose not to have nested POJO now, I think JSON schema
> input
> > > to
> > > > this utility gives more chance to expand easily later.
> > > >
> > > >
> > > >
> > > > On Fri, Jan 22, 2016 at 3:30 PM, Chinmay Kolhatkar <
> > > > chinmay@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Sure. We can make this utility to provide byte[] for compiled
> class.
> > > > > Though the utility will give a OutputStream object.. One can get
> > byte[]
> > > > > out of that.
> > > > >
> > > > > On Fri, Jan 22, 2016 at 2:56 PM, Timothy Farkas <
> tim@datatorrent.com
> > >
> > > > > wrote:
> > > > >
> > > > >> Suggestion: +1 for automatic generation of schema. The user should
> > be
> > > > able
> > > > >> to override the a manually specified schema though.
> > > > >>
> > > > >> Question: If the mechanism used to generate the class can produce
> a
> > > > byte[]
> > > > >> array, then you can send the byte[] array for the class to
> > downstream
> > > > >> operators and load the class from the byte[] array in each
> operator.
> > > An
> > > > >> example of how to do this is here:
> > > > >>
> > > > >>
> > > > >>
> > > >
> > >
> >
> http://stackoverflow.com/questions/1781091/java-how-to-load-class-stored-as-byte-into-the-jvm
> > > > >>
> > > > >> Thanks,
> > > > >> Tim
> > > > >>
> > > > >> On Fri, Jan 22, 2016 at 12:59 AM, Priyanka Gugale <
> > > > >> priyanka@datatorrent.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi,
> > > > >> >
> > > > >> > Suggestion:
> > > > >> > This came up with one of the discussion with Pramod, will it be
> a
> > > good
> > > > >> > idea, for database input operators to generate pojo based on the
> > > > >> selected
> > > > >> > column field names and types? No need to accept json input from
> > > user.
> > > > >> >
> > > > >> > Question:
> > > > >> > How can we share this POJO among multiple operators when
> > application
> > > > is
> > > > >> > already launched and class is generated on the fly?
> > > > >> >
> > > > >> > -Priyanka
> > > > >> >
> > > > >> > On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar <
> > > > >> > chinmay@datatorrent.com
> > > > >> > > wrote:
> > > > >> >
> > > > >> > > Hi All,
> > > > >> > >
> > > > >> > > We're planning to add a utility in malhar-library for
> > generating a
> > > > >> POJO
> > > > >> > > class on the fly from given JSON schema.
> > > > >> > >
> > > > >> > > Use case is where the application is provided with schema and
> > that
> > > > >> needs
> > > > >> > to
> > > > >> > > be used in one or more operators either as a tuple over the
> > stream
> > > > OR
> > > > >> for
> > > > >> > > processing.
> > > > >> > >
> > > > >> > > General Design:
> > > > >> > > 1. Utility will be provided with fqcn of the class and schema
> > > > >> definition
> > > > >> > > provided as json.
> > > > >> > > 2. The schema definition will look like following:
> > > > >> > >     {
> > > > >> > >        "fqcn":"<qualified class name>",
> > > > >> > >        "fields": [
> > > > >> > >          {
> > > > >> > >            "name":"field1",
> > > > >> > >            "type":"long"
> > > > >> > >          },
> > > > >> > >          {
> > > > >> > >            "name":"field2",
> > > > >> > >            "type": "string"
> > > > >> > >          }
> > > > >> > >        ]
> > > > >> > >     }
> > > > >> > > 3. Supported types identified in "type" JSON field are:
> > > > >> > >       boolean, char, byte, short, int, float, long, double
> > > > >> > > 4. The output of this utility will be a generated .class file
> in
> > > the
> > > > >> form
> > > > >> > > of FSDataOutputStream.
> > > > >> > > 5. Xbean asm5 library will be used for this.
> > > > >> > > 6. Following methods will be added to the generated class:
> > > > >> > >     a. Getter/Setter methods for given fields.
> > > > >> > >     b. simple toString - Generate string equivalent for all
> the
> > > > fields
> > > > >> > >     c. hashCode method - calculate the overall hashCode using
> > > > >> individual
> > > > >> > > field hashcodes, similar to how String.hashCode generates
> > hashCode
> > > > of
> > > > >> > > string using chars in it.
> > > > >> > >     d. equals method - Similar to how String.equals method has
> > > done
> > > > it
> > > > >> > > using individual char.
> > > > >> > >
> > > > >> > >
> > > > >> > > Questions I have about the functionality:
> > > > >> > > 1. Should the utility also support nested pojo definition via
> > > schema
> > > > >> > json?
> > > > >> > >     In such case the field definition can look like following:
> > > > >> > >     { "name":"nestedField", "type":"<fqcn of nested class>",
> > > > >> > > "fields":[....] }
> > > > >> > >     The nested class will be provided
> > > > >> > >
> > > > >> > > 2. Should the utility also support giving out the
> > JarOutputStream
> > > > >> which
> > > > >> > > will have a generated jar?
> > > > >> > >
> > > > >> > > Please let me know your thoughts.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Chinmay.
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Ability to generate POJO class on the fly

Posted by Priyanka Gugale <pr...@datatorrent.com>.
Say I need to emit pojo from JDBCInputOperator. I will use on fly POJO
generator for this purpose instead of providing predefined class.
Now to generate POJO on fly there are two options:
1. Get json as config from user as you specified.
2. Based on column name and type generate pojo fields and hence pojo class
without much of user configuration. (I am only considering one level here,
not the nested pojo)

Option 1 is must have to provide user a way to control how we generate
pojo. But option 2 is useful when user simply want to put column values in
some object.

-Priyanka

On Fri, Jan 22, 2016 at 6:30 PM, Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:

> Hi Priyanka,
>
> Can you give an example?
>
> Thanks,
> Chinmay.
>
> ~ Chinmay
> On 22 Jan 2016 18:14, "Priyanka Gugale" <pr...@datatorrent.com> wrote:
>
> > We do need configuration in some form like json, but I was thinking we
> can
> > have default wherever possible. This is an option and not replacement to
> > json config.
> >
> > -Priyanka
> >
> > On Fri, Jan 22, 2016 at 4:34 PM, Chinmay Kolhatkar <
> > chinmay@datatorrent.com>
> > wrote:
> >
> > > @priyanka
> > >
> > > Advantage with JSON schema is that one can represent nested POJO
> > definition
> > > in that. If you're referring to FieldInfo, I'm not sure if that is
> > possible
> > > there.
> > >
> > > Even if we chose not to have nested POJO now, I think JSON schema input
> > to
> > > this utility gives more chance to expand easily later.
> > >
> > >
> > >
> > > On Fri, Jan 22, 2016 at 3:30 PM, Chinmay Kolhatkar <
> > > chinmay@datatorrent.com>
> > > wrote:
> > >
> > > > Sure. We can make this utility to provide byte[] for compiled class.
> > > > Though the utility will give a OutputStream object.. One can get
> byte[]
> > > > out of that.
> > > >
> > > > On Fri, Jan 22, 2016 at 2:56 PM, Timothy Farkas <tim@datatorrent.com
> >
> > > > wrote:
> > > >
> > > >> Suggestion: +1 for automatic generation of schema. The user should
> be
> > > able
> > > >> to override the a manually specified schema though.
> > > >>
> > > >> Question: If the mechanism used to generate the class can produce a
> > > byte[]
> > > >> array, then you can send the byte[] array for the class to
> downstream
> > > >> operators and load the class from the byte[] array in each operator.
> > An
> > > >> example of how to do this is here:
> > > >>
> > > >>
> > > >>
> > >
> >
> http://stackoverflow.com/questions/1781091/java-how-to-load-class-stored-as-byte-into-the-jvm
> > > >>
> > > >> Thanks,
> > > >> Tim
> > > >>
> > > >> On Fri, Jan 22, 2016 at 12:59 AM, Priyanka Gugale <
> > > >> priyanka@datatorrent.com>
> > > >> wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > Suggestion:
> > > >> > This came up with one of the discussion with Pramod, will it be a
> > good
> > > >> > idea, for database input operators to generate pojo based on the
> > > >> selected
> > > >> > column field names and types? No need to accept json input from
> > user.
> > > >> >
> > > >> > Question:
> > > >> > How can we share this POJO among multiple operators when
> application
> > > is
> > > >> > already launched and class is generated on the fly?
> > > >> >
> > > >> > -Priyanka
> > > >> >
> > > >> > On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar <
> > > >> > chinmay@datatorrent.com
> > > >> > > wrote:
> > > >> >
> > > >> > > Hi All,
> > > >> > >
> > > >> > > We're planning to add a utility in malhar-library for
> generating a
> > > >> POJO
> > > >> > > class on the fly from given JSON schema.
> > > >> > >
> > > >> > > Use case is where the application is provided with schema and
> that
> > > >> needs
> > > >> > to
> > > >> > > be used in one or more operators either as a tuple over the
> stream
> > > OR
> > > >> for
> > > >> > > processing.
> > > >> > >
> > > >> > > General Design:
> > > >> > > 1. Utility will be provided with fqcn of the class and schema
> > > >> definition
> > > >> > > provided as json.
> > > >> > > 2. The schema definition will look like following:
> > > >> > >     {
> > > >> > >        "fqcn":"<qualified class name>",
> > > >> > >        "fields": [
> > > >> > >          {
> > > >> > >            "name":"field1",
> > > >> > >            "type":"long"
> > > >> > >          },
> > > >> > >          {
> > > >> > >            "name":"field2",
> > > >> > >            "type": "string"
> > > >> > >          }
> > > >> > >        ]
> > > >> > >     }
> > > >> > > 3. Supported types identified in "type" JSON field are:
> > > >> > >       boolean, char, byte, short, int, float, long, double
> > > >> > > 4. The output of this utility will be a generated .class file in
> > the
> > > >> form
> > > >> > > of FSDataOutputStream.
> > > >> > > 5. Xbean asm5 library will be used for this.
> > > >> > > 6. Following methods will be added to the generated class:
> > > >> > >     a. Getter/Setter methods for given fields.
> > > >> > >     b. simple toString - Generate string equivalent for all the
> > > fields
> > > >> > >     c. hashCode method - calculate the overall hashCode using
> > > >> individual
> > > >> > > field hashcodes, similar to how String.hashCode generates
> hashCode
> > > of
> > > >> > > string using chars in it.
> > > >> > >     d. equals method - Similar to how String.equals method has
> > done
> > > it
> > > >> > > using individual char.
> > > >> > >
> > > >> > >
> > > >> > > Questions I have about the functionality:
> > > >> > > 1. Should the utility also support nested pojo definition via
> > schema
> > > >> > json?
> > > >> > >     In such case the field definition can look like following:
> > > >> > >     { "name":"nestedField", "type":"<fqcn of nested class>",
> > > >> > > "fields":[....] }
> > > >> > >     The nested class will be provided
> > > >> > >
> > > >> > > 2. Should the utility also support giving out the
> JarOutputStream
> > > >> which
> > > >> > > will have a generated jar?
> > > >> > >
> > > >> > > Please let me know your thoughts.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Chinmay.
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: Ability to generate POJO class on the fly

Posted by Bhupesh Chawda <bh...@datatorrent.com>.
I think this will utilize how the JSON schema to Class FSDataOutputStream
is implemented.
The JSON would have some internal representation in a class. I think we can
also use that internal representation to have the class generated rather
than providing the JSON string strictly everytime.

Something like: JSON String -> Internal Java Object -> Generated Class (as
FSDataOutputStream)

Thanks.
-Bhupesh


On Fri, Jan 22, 2016 at 6:30 PM, Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:

> Hi Priyanka,
>
> Can you give an example?
>
> Thanks,
> Chinmay.
>
> ~ Chinmay
> On 22 Jan 2016 18:14, "Priyanka Gugale" <pr...@datatorrent.com> wrote:
>
> > We do need configuration in some form like json, but I was thinking we
> can
> > have default wherever possible. This is an option and not replacement to
> > json config.
> >
> > -Priyanka
> >
> > On Fri, Jan 22, 2016 at 4:34 PM, Chinmay Kolhatkar <
> > chinmay@datatorrent.com>
> > wrote:
> >
> > > @priyanka
> > >
> > > Advantage with JSON schema is that one can represent nested POJO
> > definition
> > > in that. If you're referring to FieldInfo, I'm not sure if that is
> > possible
> > > there.
> > >
> > > Even if we chose not to have nested POJO now, I think JSON schema input
> > to
> > > this utility gives more chance to expand easily later.
> > >
> > >
> > >
> > > On Fri, Jan 22, 2016 at 3:30 PM, Chinmay Kolhatkar <
> > > chinmay@datatorrent.com>
> > > wrote:
> > >
> > > > Sure. We can make this utility to provide byte[] for compiled class.
> > > > Though the utility will give a OutputStream object.. One can get
> byte[]
> > > > out of that.
> > > >
> > > > On Fri, Jan 22, 2016 at 2:56 PM, Timothy Farkas <tim@datatorrent.com
> >
> > > > wrote:
> > > >
> > > >> Suggestion: +1 for automatic generation of schema. The user should
> be
> > > able
> > > >> to override the a manually specified schema though.
> > > >>
> > > >> Question: If the mechanism used to generate the class can produce a
> > > byte[]
> > > >> array, then you can send the byte[] array for the class to
> downstream
> > > >> operators and load the class from the byte[] array in each operator.
> > An
> > > >> example of how to do this is here:
> > > >>
> > > >>
> > > >>
> > >
> >
> http://stackoverflow.com/questions/1781091/java-how-to-load-class-stored-as-byte-into-the-jvm
> > > >>
> > > >> Thanks,
> > > >> Tim
> > > >>
> > > >> On Fri, Jan 22, 2016 at 12:59 AM, Priyanka Gugale <
> > > >> priyanka@datatorrent.com>
> > > >> wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > Suggestion:
> > > >> > This came up with one of the discussion with Pramod, will it be a
> > good
> > > >> > idea, for database input operators to generate pojo based on the
> > > >> selected
> > > >> > column field names and types? No need to accept json input from
> > user.
> > > >> >
> > > >> > Question:
> > > >> > How can we share this POJO among multiple operators when
> application
> > > is
> > > >> > already launched and class is generated on the fly?
> > > >> >
> > > >> > -Priyanka
> > > >> >
> > > >> > On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar <
> > > >> > chinmay@datatorrent.com
> > > >> > > wrote:
> > > >> >
> > > >> > > Hi All,
> > > >> > >
> > > >> > > We're planning to add a utility in malhar-library for
> generating a
> > > >> POJO
> > > >> > > class on the fly from given JSON schema.
> > > >> > >
> > > >> > > Use case is where the application is provided with schema and
> that
> > > >> needs
> > > >> > to
> > > >> > > be used in one or more operators either as a tuple over the
> stream
> > > OR
> > > >> for
> > > >> > > processing.
> > > >> > >
> > > >> > > General Design:
> > > >> > > 1. Utility will be provided with fqcn of the class and schema
> > > >> definition
> > > >> > > provided as json.
> > > >> > > 2. The schema definition will look like following:
> > > >> > >     {
> > > >> > >        "fqcn":"<qualified class name>",
> > > >> > >        "fields": [
> > > >> > >          {
> > > >> > >            "name":"field1",
> > > >> > >            "type":"long"
> > > >> > >          },
> > > >> > >          {
> > > >> > >            "name":"field2",
> > > >> > >            "type": "string"
> > > >> > >          }
> > > >> > >        ]
> > > >> > >     }
> > > >> > > 3. Supported types identified in "type" JSON field are:
> > > >> > >       boolean, char, byte, short, int, float, long, double
> > > >> > > 4. The output of this utility will be a generated .class file in
> > the
> > > >> form
> > > >> > > of FSDataOutputStream.
> > > >> > > 5. Xbean asm5 library will be used for this.
> > > >> > > 6. Following methods will be added to the generated class:
> > > >> > >     a. Getter/Setter methods for given fields.
> > > >> > >     b. simple toString - Generate string equivalent for all the
> > > fields
> > > >> > >     c. hashCode method - calculate the overall hashCode using
> > > >> individual
> > > >> > > field hashcodes, similar to how String.hashCode generates
> hashCode
> > > of
> > > >> > > string using chars in it.
> > > >> > >     d. equals method - Similar to how String.equals method has
> > done
> > > it
> > > >> > > using individual char.
> > > >> > >
> > > >> > >
> > > >> > > Questions I have about the functionality:
> > > >> > > 1. Should the utility also support nested pojo definition via
> > schema
> > > >> > json?
> > > >> > >     In such case the field definition can look like following:
> > > >> > >     { "name":"nestedField", "type":"<fqcn of nested class>",
> > > >> > > "fields":[....] }
> > > >> > >     The nested class will be provided
> > > >> > >
> > > >> > > 2. Should the utility also support giving out the
> JarOutputStream
> > > >> which
> > > >> > > will have a generated jar?
> > > >> > >
> > > >> > > Please let me know your thoughts.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Chinmay.
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: Ability to generate POJO class on the fly

Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
Hi Priyanka,

Can you give an example?

Thanks,
Chinmay.

~ Chinmay
On 22 Jan 2016 18:14, "Priyanka Gugale" <pr...@datatorrent.com> wrote:

> We do need configuration in some form like json, but I was thinking we can
> have default wherever possible. This is an option and not replacement to
> json config.
>
> -Priyanka
>
> On Fri, Jan 22, 2016 at 4:34 PM, Chinmay Kolhatkar <
> chinmay@datatorrent.com>
> wrote:
>
> > @priyanka
> >
> > Advantage with JSON schema is that one can represent nested POJO
> definition
> > in that. If you're referring to FieldInfo, I'm not sure if that is
> possible
> > there.
> >
> > Even if we chose not to have nested POJO now, I think JSON schema input
> to
> > this utility gives more chance to expand easily later.
> >
> >
> >
> > On Fri, Jan 22, 2016 at 3:30 PM, Chinmay Kolhatkar <
> > chinmay@datatorrent.com>
> > wrote:
> >
> > > Sure. We can make this utility to provide byte[] for compiled class.
> > > Though the utility will give a OutputStream object.. One can get byte[]
> > > out of that.
> > >
> > > On Fri, Jan 22, 2016 at 2:56 PM, Timothy Farkas <ti...@datatorrent.com>
> > > wrote:
> > >
> > >> Suggestion: +1 for automatic generation of schema. The user should be
> > able
> > >> to override the a manually specified schema though.
> > >>
> > >> Question: If the mechanism used to generate the class can produce a
> > byte[]
> > >> array, then you can send the byte[] array for the class to downstream
> > >> operators and load the class from the byte[] array in each operator.
> An
> > >> example of how to do this is here:
> > >>
> > >>
> > >>
> >
> http://stackoverflow.com/questions/1781091/java-how-to-load-class-stored-as-byte-into-the-jvm
> > >>
> > >> Thanks,
> > >> Tim
> > >>
> > >> On Fri, Jan 22, 2016 at 12:59 AM, Priyanka Gugale <
> > >> priyanka@datatorrent.com>
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > Suggestion:
> > >> > This came up with one of the discussion with Pramod, will it be a
> good
> > >> > idea, for database input operators to generate pojo based on the
> > >> selected
> > >> > column field names and types? No need to accept json input from
> user.
> > >> >
> > >> > Question:
> > >> > How can we share this POJO among multiple operators when application
> > is
> > >> > already launched and class is generated on the fly?
> > >> >
> > >> > -Priyanka
> > >> >
> > >> > On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar <
> > >> > chinmay@datatorrent.com
> > >> > > wrote:
> > >> >
> > >> > > Hi All,
> > >> > >
> > >> > > We're planning to add a utility in malhar-library for generating a
> > >> POJO
> > >> > > class on the fly from given JSON schema.
> > >> > >
> > >> > > Use case is where the application is provided with schema and that
> > >> needs
> > >> > to
> > >> > > be used in one or more operators either as a tuple over the stream
> > OR
> > >> for
> > >> > > processing.
> > >> > >
> > >> > > General Design:
> > >> > > 1. Utility will be provided with fqcn of the class and schema
> > >> definition
> > >> > > provided as json.
> > >> > > 2. The schema definition will look like following:
> > >> > >     {
> > >> > >        "fqcn":"<qualified class name>",
> > >> > >        "fields": [
> > >> > >          {
> > >> > >            "name":"field1",
> > >> > >            "type":"long"
> > >> > >          },
> > >> > >          {
> > >> > >            "name":"field2",
> > >> > >            "type": "string"
> > >> > >          }
> > >> > >        ]
> > >> > >     }
> > >> > > 3. Supported types identified in "type" JSON field are:
> > >> > >       boolean, char, byte, short, int, float, long, double
> > >> > > 4. The output of this utility will be a generated .class file in
> the
> > >> form
> > >> > > of FSDataOutputStream.
> > >> > > 5. Xbean asm5 library will be used for this.
> > >> > > 6. Following methods will be added to the generated class:
> > >> > >     a. Getter/Setter methods for given fields.
> > >> > >     b. simple toString - Generate string equivalent for all the
> > fields
> > >> > >     c. hashCode method - calculate the overall hashCode using
> > >> individual
> > >> > > field hashcodes, similar to how String.hashCode generates hashCode
> > of
> > >> > > string using chars in it.
> > >> > >     d. equals method - Similar to how String.equals method has
> done
> > it
> > >> > > using individual char.
> > >> > >
> > >> > >
> > >> > > Questions I have about the functionality:
> > >> > > 1. Should the utility also support nested pojo definition via
> schema
> > >> > json?
> > >> > >     In such case the field definition can look like following:
> > >> > >     { "name":"nestedField", "type":"<fqcn of nested class>",
> > >> > > "fields":[....] }
> > >> > >     The nested class will be provided
> > >> > >
> > >> > > 2. Should the utility also support giving out the JarOutputStream
> > >> which
> > >> > > will have a generated jar?
> > >> > >
> > >> > > Please let me know your thoughts.
> > >> > >
> > >> > > Thanks,
> > >> > > Chinmay.
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: Ability to generate POJO class on the fly

Posted by Priyanka Gugale <pr...@datatorrent.com>.
We do need configuration in some form like json, but I was thinking we can
have default wherever possible. This is an option and not replacement to
json config.

-Priyanka

On Fri, Jan 22, 2016 at 4:34 PM, Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:

> @priyanka
>
> Advantage with JSON schema is that one can represent nested POJO definition
> in that. If you're referring to FieldInfo, I'm not sure if that is possible
> there.
>
> Even if we chose not to have nested POJO now, I think JSON schema input to
> this utility gives more chance to expand easily later.
>
>
>
> On Fri, Jan 22, 2016 at 3:30 PM, Chinmay Kolhatkar <
> chinmay@datatorrent.com>
> wrote:
>
> > Sure. We can make this utility to provide byte[] for compiled class.
> > Though the utility will give a OutputStream object.. One can get byte[]
> > out of that.
> >
> > On Fri, Jan 22, 2016 at 2:56 PM, Timothy Farkas <ti...@datatorrent.com>
> > wrote:
> >
> >> Suggestion: +1 for automatic generation of schema. The user should be
> able
> >> to override the a manually specified schema though.
> >>
> >> Question: If the mechanism used to generate the class can produce a
> byte[]
> >> array, then you can send the byte[] array for the class to downstream
> >> operators and load the class from the byte[] array in each operator. An
> >> example of how to do this is here:
> >>
> >>
> >>
> http://stackoverflow.com/questions/1781091/java-how-to-load-class-stored-as-byte-into-the-jvm
> >>
> >> Thanks,
> >> Tim
> >>
> >> On Fri, Jan 22, 2016 at 12:59 AM, Priyanka Gugale <
> >> priyanka@datatorrent.com>
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > Suggestion:
> >> > This came up with one of the discussion with Pramod, will it be a good
> >> > idea, for database input operators to generate pojo based on the
> >> selected
> >> > column field names and types? No need to accept json input from user.
> >> >
> >> > Question:
> >> > How can we share this POJO among multiple operators when application
> is
> >> > already launched and class is generated on the fly?
> >> >
> >> > -Priyanka
> >> >
> >> > On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar <
> >> > chinmay@datatorrent.com
> >> > > wrote:
> >> >
> >> > > Hi All,
> >> > >
> >> > > We're planning to add a utility in malhar-library for generating a
> >> POJO
> >> > > class on the fly from given JSON schema.
> >> > >
> >> > > Use case is where the application is provided with schema and that
> >> needs
> >> > to
> >> > > be used in one or more operators either as a tuple over the stream
> OR
> >> for
> >> > > processing.
> >> > >
> >> > > General Design:
> >> > > 1. Utility will be provided with fqcn of the class and schema
> >> definition
> >> > > provided as json.
> >> > > 2. The schema definition will look like following:
> >> > >     {
> >> > >        "fqcn":"<qualified class name>",
> >> > >        "fields": [
> >> > >          {
> >> > >            "name":"field1",
> >> > >            "type":"long"
> >> > >          },
> >> > >          {
> >> > >            "name":"field2",
> >> > >            "type": "string"
> >> > >          }
> >> > >        ]
> >> > >     }
> >> > > 3. Supported types identified in "type" JSON field are:
> >> > >       boolean, char, byte, short, int, float, long, double
> >> > > 4. The output of this utility will be a generated .class file in the
> >> form
> >> > > of FSDataOutputStream.
> >> > > 5. Xbean asm5 library will be used for this.
> >> > > 6. Following methods will be added to the generated class:
> >> > >     a. Getter/Setter methods for given fields.
> >> > >     b. simple toString - Generate string equivalent for all the
> fields
> >> > >     c. hashCode method - calculate the overall hashCode using
> >> individual
> >> > > field hashcodes, similar to how String.hashCode generates hashCode
> of
> >> > > string using chars in it.
> >> > >     d. equals method - Similar to how String.equals method has done
> it
> >> > > using individual char.
> >> > >
> >> > >
> >> > > Questions I have about the functionality:
> >> > > 1. Should the utility also support nested pojo definition via schema
> >> > json?
> >> > >     In such case the field definition can look like following:
> >> > >     { "name":"nestedField", "type":"<fqcn of nested class>",
> >> > > "fields":[....] }
> >> > >     The nested class will be provided
> >> > >
> >> > > 2. Should the utility also support giving out the JarOutputStream
> >> which
> >> > > will have a generated jar?
> >> > >
> >> > > Please let me know your thoughts.
> >> > >
> >> > > Thanks,
> >> > > Chinmay.
> >> > >
> >> >
> >>
> >
> >
>

Re: Ability to generate POJO class on the fly

Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
@priyanka

Advantage with JSON schema is that one can represent nested POJO definition
in that. If you're referring to FieldInfo, I'm not sure if that is possible
there.

Even if we chose not to have nested POJO now, I think JSON schema input to
this utility gives more chance to expand easily later.



On Fri, Jan 22, 2016 at 3:30 PM, Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:

> Sure. We can make this utility to provide byte[] for compiled class.
> Though the utility will give a OutputStream object.. One can get byte[]
> out of that.
>
> On Fri, Jan 22, 2016 at 2:56 PM, Timothy Farkas <ti...@datatorrent.com>
> wrote:
>
>> Suggestion: +1 for automatic generation of schema. The user should be able
>> to override the a manually specified schema though.
>>
>> Question: If the mechanism used to generate the class can produce a byte[]
>> array, then you can send the byte[] array for the class to downstream
>> operators and load the class from the byte[] array in each operator. An
>> example of how to do this is here:
>>
>>
>> http://stackoverflow.com/questions/1781091/java-how-to-load-class-stored-as-byte-into-the-jvm
>>
>> Thanks,
>> Tim
>>
>> On Fri, Jan 22, 2016 at 12:59 AM, Priyanka Gugale <
>> priyanka@datatorrent.com>
>> wrote:
>>
>> > Hi,
>> >
>> > Suggestion:
>> > This came up with one of the discussion with Pramod, will it be a good
>> > idea, for database input operators to generate pojo based on the
>> selected
>> > column field names and types? No need to accept json input from user.
>> >
>> > Question:
>> > How can we share this POJO among multiple operators when application is
>> > already launched and class is generated on the fly?
>> >
>> > -Priyanka
>> >
>> > On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar <
>> > chinmay@datatorrent.com
>> > > wrote:
>> >
>> > > Hi All,
>> > >
>> > > We're planning to add a utility in malhar-library for generating a
>> POJO
>> > > class on the fly from given JSON schema.
>> > >
>> > > Use case is where the application is provided with schema and that
>> needs
>> > to
>> > > be used in one or more operators either as a tuple over the stream OR
>> for
>> > > processing.
>> > >
>> > > General Design:
>> > > 1. Utility will be provided with fqcn of the class and schema
>> definition
>> > > provided as json.
>> > > 2. The schema definition will look like following:
>> > >     {
>> > >        "fqcn":"<qualified class name>",
>> > >        "fields": [
>> > >          {
>> > >            "name":"field1",
>> > >            "type":"long"
>> > >          },
>> > >          {
>> > >            "name":"field2",
>> > >            "type": "string"
>> > >          }
>> > >        ]
>> > >     }
>> > > 3. Supported types identified in "type" JSON field are:
>> > >       boolean, char, byte, short, int, float, long, double
>> > > 4. The output of this utility will be a generated .class file in the
>> form
>> > > of FSDataOutputStream.
>> > > 5. Xbean asm5 library will be used for this.
>> > > 6. Following methods will be added to the generated class:
>> > >     a. Getter/Setter methods for given fields.
>> > >     b. simple toString - Generate string equivalent for all the fields
>> > >     c. hashCode method - calculate the overall hashCode using
>> individual
>> > > field hashcodes, similar to how String.hashCode generates hashCode of
>> > > string using chars in it.
>> > >     d. equals method - Similar to how String.equals method has done it
>> > > using individual char.
>> > >
>> > >
>> > > Questions I have about the functionality:
>> > > 1. Should the utility also support nested pojo definition via schema
>> > json?
>> > >     In such case the field definition can look like following:
>> > >     { "name":"nestedField", "type":"<fqcn of nested class>",
>> > > "fields":[....] }
>> > >     The nested class will be provided
>> > >
>> > > 2. Should the utility also support giving out the JarOutputStream
>> which
>> > > will have a generated jar?
>> > >
>> > > Please let me know your thoughts.
>> > >
>> > > Thanks,
>> > > Chinmay.
>> > >
>> >
>>
>
>

Re: Ability to generate POJO class on the fly

Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
Sure. We can make this utility to provide byte[] for compiled class.
Though the utility will give a OutputStream object.. One can get byte[] out
of that.

On Fri, Jan 22, 2016 at 2:56 PM, Timothy Farkas <ti...@datatorrent.com> wrote:

> Suggestion: +1 for automatic generation of schema. The user should be able
> to override the a manually specified schema though.
>
> Question: If the mechanism used to generate the class can produce a byte[]
> array, then you can send the byte[] array for the class to downstream
> operators and load the class from the byte[] array in each operator. An
> example of how to do this is here:
>
>
> http://stackoverflow.com/questions/1781091/java-how-to-load-class-stored-as-byte-into-the-jvm
>
> Thanks,
> Tim
>
> On Fri, Jan 22, 2016 at 12:59 AM, Priyanka Gugale <
> priyanka@datatorrent.com>
> wrote:
>
> > Hi,
> >
> > Suggestion:
> > This came up with one of the discussion with Pramod, will it be a good
> > idea, for database input operators to generate pojo based on the selected
> > column field names and types? No need to accept json input from user.
> >
> > Question:
> > How can we share this POJO among multiple operators when application is
> > already launched and class is generated on the fly?
> >
> > -Priyanka
> >
> > On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar <
> > chinmay@datatorrent.com
> > > wrote:
> >
> > > Hi All,
> > >
> > > We're planning to add a utility in malhar-library for generating a POJO
> > > class on the fly from given JSON schema.
> > >
> > > Use case is where the application is provided with schema and that
> needs
> > to
> > > be used in one or more operators either as a tuple over the stream OR
> for
> > > processing.
> > >
> > > General Design:
> > > 1. Utility will be provided with fqcn of the class and schema
> definition
> > > provided as json.
> > > 2. The schema definition will look like following:
> > >     {
> > >        "fqcn":"<qualified class name>",
> > >        "fields": [
> > >          {
> > >            "name":"field1",
> > >            "type":"long"
> > >          },
> > >          {
> > >            "name":"field2",
> > >            "type": "string"
> > >          }
> > >        ]
> > >     }
> > > 3. Supported types identified in "type" JSON field are:
> > >       boolean, char, byte, short, int, float, long, double
> > > 4. The output of this utility will be a generated .class file in the
> form
> > > of FSDataOutputStream.
> > > 5. Xbean asm5 library will be used for this.
> > > 6. Following methods will be added to the generated class:
> > >     a. Getter/Setter methods for given fields.
> > >     b. simple toString - Generate string equivalent for all the fields
> > >     c. hashCode method - calculate the overall hashCode using
> individual
> > > field hashcodes, similar to how String.hashCode generates hashCode of
> > > string using chars in it.
> > >     d. equals method - Similar to how String.equals method has done it
> > > using individual char.
> > >
> > >
> > > Questions I have about the functionality:
> > > 1. Should the utility also support nested pojo definition via schema
> > json?
> > >     In such case the field definition can look like following:
> > >     { "name":"nestedField", "type":"<fqcn of nested class>",
> > > "fields":[....] }
> > >     The nested class will be provided
> > >
> > > 2. Should the utility also support giving out the JarOutputStream which
> > > will have a generated jar?
> > >
> > > Please let me know your thoughts.
> > >
> > > Thanks,
> > > Chinmay.
> > >
> >
>

Re: Ability to generate POJO class on the fly

Posted by Timothy Farkas <ti...@datatorrent.com>.
Suggestion: +1 for automatic generation of schema. The user should be able
to override the a manually specified schema though.

Question: If the mechanism used to generate the class can produce a byte[]
array, then you can send the byte[] array for the class to downstream
operators and load the class from the byte[] array in each operator. An
example of how to do this is here:

http://stackoverflow.com/questions/1781091/java-how-to-load-class-stored-as-byte-into-the-jvm

Thanks,
Tim

On Fri, Jan 22, 2016 at 12:59 AM, Priyanka Gugale <pr...@datatorrent.com>
wrote:

> Hi,
>
> Suggestion:
> This came up with one of the discussion with Pramod, will it be a good
> idea, for database input operators to generate pojo based on the selected
> column field names and types? No need to accept json input from user.
>
> Question:
> How can we share this POJO among multiple operators when application is
> already launched and class is generated on the fly?
>
> -Priyanka
>
> On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar <
> chinmay@datatorrent.com
> > wrote:
>
> > Hi All,
> >
> > We're planning to add a utility in malhar-library for generating a POJO
> > class on the fly from given JSON schema.
> >
> > Use case is where the application is provided with schema and that needs
> to
> > be used in one or more operators either as a tuple over the stream OR for
> > processing.
> >
> > General Design:
> > 1. Utility will be provided with fqcn of the class and schema definition
> > provided as json.
> > 2. The schema definition will look like following:
> >     {
> >        "fqcn":"<qualified class name>",
> >        "fields": [
> >          {
> >            "name":"field1",
> >            "type":"long"
> >          },
> >          {
> >            "name":"field2",
> >            "type": "string"
> >          }
> >        ]
> >     }
> > 3. Supported types identified in "type" JSON field are:
> >       boolean, char, byte, short, int, float, long, double
> > 4. The output of this utility will be a generated .class file in the form
> > of FSDataOutputStream.
> > 5. Xbean asm5 library will be used for this.
> > 6. Following methods will be added to the generated class:
> >     a. Getter/Setter methods for given fields.
> >     b. simple toString - Generate string equivalent for all the fields
> >     c. hashCode method - calculate the overall hashCode using individual
> > field hashcodes, similar to how String.hashCode generates hashCode of
> > string using chars in it.
> >     d. equals method - Similar to how String.equals method has done it
> > using individual char.
> >
> >
> > Questions I have about the functionality:
> > 1. Should the utility also support nested pojo definition via schema
> json?
> >     In such case the field definition can look like following:
> >     { "name":"nestedField", "type":"<fqcn of nested class>",
> > "fields":[....] }
> >     The nested class will be provided
> >
> > 2. Should the utility also support giving out the JarOutputStream which
> > will have a generated jar?
> >
> > Please let me know your thoughts.
> >
> > Thanks,
> > Chinmay.
> >
>

Re: Ability to generate POJO class on the fly

Posted by Priyanka Gugale <pr...@datatorrent.com>.
Hi,

Suggestion:
This came up with one of the discussion with Pramod, will it be a good
idea, for database input operators to generate pojo based on the selected
column field names and types? No need to accept json input from user.

Question:
How can we share this POJO among multiple operators when application is
already launched and class is generated on the fly?

-Priyanka

On Thu, Jan 21, 2016 at 11:16 PM, Chinmay Kolhatkar <chinmay@datatorrent.com
> wrote:

> Hi All,
>
> We're planning to add a utility in malhar-library for generating a POJO
> class on the fly from given JSON schema.
>
> Use case is where the application is provided with schema and that needs to
> be used in one or more operators either as a tuple over the stream OR for
> processing.
>
> General Design:
> 1. Utility will be provided with fqcn of the class and schema definition
> provided as json.
> 2. The schema definition will look like following:
>     {
>        "fqcn":"<qualified class name>",
>        "fields": [
>          {
>            "name":"field1",
>            "type":"long"
>          },
>          {
>            "name":"field2",
>            "type": "string"
>          }
>        ]
>     }
> 3. Supported types identified in "type" JSON field are:
>       boolean, char, byte, short, int, float, long, double
> 4. The output of this utility will be a generated .class file in the form
> of FSDataOutputStream.
> 5. Xbean asm5 library will be used for this.
> 6. Following methods will be added to the generated class:
>     a. Getter/Setter methods for given fields.
>     b. simple toString - Generate string equivalent for all the fields
>     c. hashCode method - calculate the overall hashCode using individual
> field hashcodes, similar to how String.hashCode generates hashCode of
> string using chars in it.
>     d. equals method - Similar to how String.equals method has done it
> using individual char.
>
>
> Questions I have about the functionality:
> 1. Should the utility also support nested pojo definition via schema json?
>     In such case the field definition can look like following:
>     { "name":"nestedField", "type":"<fqcn of nested class>",
> "fields":[....] }
>     The nested class will be provided
>
> 2. Should the utility also support giving out the JarOutputStream which
> will have a generated jar?
>
> Please let me know your thoughts.
>
> Thanks,
> Chinmay.
>