You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metamodel.apache.org by Kasper Sørensen <i....@gmail.com> on 2014/07/03 12:26:54 UTC
[DISCUSS] JSON DataContext and a 'Schema inference' aspect in the
'core' module
Hi guys,
Maybe you saw that I posted a review request [1] yesterday for METAMODEL-38
(a JSON based module for MetaModel).
I was building this JSON module and trying to do it in a way where the user
could configure how the logical schema would look like. In some cases you
would want MetaModel to infer the schema based on a sample of documents in
the source, and in other cases you might want MetaModel to just treat the
source as a 1-column table with a MAP data type. There's probably also
other strategies.
That part I felt was also very relevant for many other "schemaless"
datastores, such as MongoDB, CouchDB, HBase etc. So I put there interfaces
and a few standard implementations of it into the core module, and applied
it to the JSON module. If this idea is accepted, I would like to also add
it to MongoDB and CouchDB modules (those are a natural fit) and maybe also
HBase (slight more advanced because of the column-family concept).
I think it makes sense to open a DISCUSS thread about this approach, since
Schema Inference is in itself a very nice distinguishing feature I think.
I'd like to invite anyone to share their ideas here, so that this is maybe
a place where we can make MetaModel shine.
Cheers,
Kasper
[1] https://reviews.apache.org/r/23228/
Re: [DISCUSS] JSON DataContext and a 'Schema inference' aspect in the
'core' module
Posted by Kasper Sørensen <i....@gmail.com>.
I did quite a lot of changes yesterday evening because I was trying to
apply the same principles to the CouchDB module. The idea is still the
same, but the SchemaBuilder interface and related classes have changed
quite a lot, because it was obviously not going to work with a multi-table
approach like in CouchDB. So please also take into consideration this
review request: https://reviews.apache.org/r/23377
2014-07-09 19:23 GMT+02:00 Kasper Sørensen <i....@gmail.com>:
> I'll go ahead then, noone seems to have an issue with this :-)
>
>
> 2014-07-08 18:29 GMT+02:00 Ankit Kumar <ak...@gmail.com>:
>
> +1
>>
>> Ankit
>>
>> > On 08 Jul 2014, at 15:09, Kasper Sørensen <
>> i.am.kasper.sorensen@gmail.com> wrote:
>> >
>> > Well almost. It would be a Map<String,Object>, since the value-type is
>> > unknown. For instance, if the document had nested objects or arrays, or
>> > even a number-value, it would not always be a string value. The metadata
>> > would say that the column type is ColumnType.MAP
>> >
>> >
>> > 2014-07-08 12:56 GMT+02:00 Ankit Kumar <ak...@gmail.com>:
>> >
>> >> Hi Kasper,
>> >>
>> >> Quick question - Will the map based interpretation of the JSON be a
>> >> Map<String,String> type and would we get this automatically in that
>> type
>> >> when reading the column from the row.
>> >>
>> >> If that's the idea then I would vote for it [+1] as we can already
>> benefit
>> >> from this.
>> >>
>> >> If I interpret it incorrect then do please enlighten me.
>> >>
>> >> Regards
>> >> Ankit
>> >>
>> >>
>> >> On Thu, Jul 3, 2014 at 12:26 PM, Kasper Sørensen <
>> >> i.am.kasper.sorensen@gmail.com> wrote:
>> >>
>> >>> Hi guys,
>> >>>
>> >>> Maybe you saw that I posted a review request [1] yesterday for
>> >> METAMODEL-38
>> >>> (a JSON based module for MetaModel).
>> >>>
>> >>> I was building this JSON module and trying to do it in a way where the
>> >> user
>> >>> could configure how the logical schema would look like. In some cases
>> you
>> >>> would want MetaModel to infer the schema based on a sample of
>> documents
>> >> in
>> >>> the source, and in other cases you might want MetaModel to just treat
>> the
>> >>> source as a 1-column table with a MAP data type. There's probably also
>> >>> other strategies.
>> >>>
>> >>> That part I felt was also very relevant for many other "schemaless"
>> >>> datastores, such as MongoDB, CouchDB, HBase etc. So I put there
>> >> interfaces
>> >>> and a few standard implementations of it into the core module, and
>> >> applied
>> >>> it to the JSON module. If this idea is accepted, I would like to also
>> add
>> >>> it to MongoDB and CouchDB modules (those are a natural fit) and maybe
>> >> also
>> >>> HBase (slight more advanced because of the column-family concept).
>> >>>
>> >>> I think it makes sense to open a DISCUSS thread about this approach,
>> >> since
>> >>> Schema Inference is in itself a very nice distinguishing feature I
>> think.
>> >>> I'd like to invite anyone to share their ideas here, so that this is
>> >> maybe
>> >>> a place where we can make MetaModel shine.
>> >>>
>> >>> Cheers,
>> >>> Kasper
>> >>>
>> >>> [1] https://reviews.apache.org/r/23228/
>> >>
>>
>
>
Re: [DISCUSS] JSON DataContext and a 'Schema inference' aspect in the
'core' module
Posted by Kasper Sørensen <i....@gmail.com>.
I'll go ahead then, noone seems to have an issue with this :-)
2014-07-08 18:29 GMT+02:00 Ankit Kumar <ak...@gmail.com>:
> +1
>
> Ankit
>
> > On 08 Jul 2014, at 15:09, Kasper Sørensen <
> i.am.kasper.sorensen@gmail.com> wrote:
> >
> > Well almost. It would be a Map<String,Object>, since the value-type is
> > unknown. For instance, if the document had nested objects or arrays, or
> > even a number-value, it would not always be a string value. The metadata
> > would say that the column type is ColumnType.MAP
> >
> >
> > 2014-07-08 12:56 GMT+02:00 Ankit Kumar <ak...@gmail.com>:
> >
> >> Hi Kasper,
> >>
> >> Quick question - Will the map based interpretation of the JSON be a
> >> Map<String,String> type and would we get this automatically in that type
> >> when reading the column from the row.
> >>
> >> If that's the idea then I would vote for it [+1] as we can already
> benefit
> >> from this.
> >>
> >> If I interpret it incorrect then do please enlighten me.
> >>
> >> Regards
> >> Ankit
> >>
> >>
> >> On Thu, Jul 3, 2014 at 12:26 PM, Kasper Sørensen <
> >> i.am.kasper.sorensen@gmail.com> wrote:
> >>
> >>> Hi guys,
> >>>
> >>> Maybe you saw that I posted a review request [1] yesterday for
> >> METAMODEL-38
> >>> (a JSON based module for MetaModel).
> >>>
> >>> I was building this JSON module and trying to do it in a way where the
> >> user
> >>> could configure how the logical schema would look like. In some cases
> you
> >>> would want MetaModel to infer the schema based on a sample of documents
> >> in
> >>> the source, and in other cases you might want MetaModel to just treat
> the
> >>> source as a 1-column table with a MAP data type. There's probably also
> >>> other strategies.
> >>>
> >>> That part I felt was also very relevant for many other "schemaless"
> >>> datastores, such as MongoDB, CouchDB, HBase etc. So I put there
> >> interfaces
> >>> and a few standard implementations of it into the core module, and
> >> applied
> >>> it to the JSON module. If this idea is accepted, I would like to also
> add
> >>> it to MongoDB and CouchDB modules (those are a natural fit) and maybe
> >> also
> >>> HBase (slight more advanced because of the column-family concept).
> >>>
> >>> I think it makes sense to open a DISCUSS thread about this approach,
> >> since
> >>> Schema Inference is in itself a very nice distinguishing feature I
> think.
> >>> I'd like to invite anyone to share their ideas here, so that this is
> >> maybe
> >>> a place where we can make MetaModel shine.
> >>>
> >>> Cheers,
> >>> Kasper
> >>>
> >>> [1] https://reviews.apache.org/r/23228/
> >>
>
Re: [DISCUSS] JSON DataContext and a 'Schema inference' aspect in the 'core' module
Posted by Ankit Kumar <ak...@gmail.com>.
+1
Ankit
> On 08 Jul 2014, at 15:09, Kasper Sørensen <i....@gmail.com> wrote:
>
> Well almost. It would be a Map<String,Object>, since the value-type is
> unknown. For instance, if the document had nested objects or arrays, or
> even a number-value, it would not always be a string value. The metadata
> would say that the column type is ColumnType.MAP
>
>
> 2014-07-08 12:56 GMT+02:00 Ankit Kumar <ak...@gmail.com>:
>
>> Hi Kasper,
>>
>> Quick question - Will the map based interpretation of the JSON be a
>> Map<String,String> type and would we get this automatically in that type
>> when reading the column from the row.
>>
>> If that's the idea then I would vote for it [+1] as we can already benefit
>> from this.
>>
>> If I interpret it incorrect then do please enlighten me.
>>
>> Regards
>> Ankit
>>
>>
>> On Thu, Jul 3, 2014 at 12:26 PM, Kasper Sørensen <
>> i.am.kasper.sorensen@gmail.com> wrote:
>>
>>> Hi guys,
>>>
>>> Maybe you saw that I posted a review request [1] yesterday for
>> METAMODEL-38
>>> (a JSON based module for MetaModel).
>>>
>>> I was building this JSON module and trying to do it in a way where the
>> user
>>> could configure how the logical schema would look like. In some cases you
>>> would want MetaModel to infer the schema based on a sample of documents
>> in
>>> the source, and in other cases you might want MetaModel to just treat the
>>> source as a 1-column table with a MAP data type. There's probably also
>>> other strategies.
>>>
>>> That part I felt was also very relevant for many other "schemaless"
>>> datastores, such as MongoDB, CouchDB, HBase etc. So I put there
>> interfaces
>>> and a few standard implementations of it into the core module, and
>> applied
>>> it to the JSON module. If this idea is accepted, I would like to also add
>>> it to MongoDB and CouchDB modules (those are a natural fit) and maybe
>> also
>>> HBase (slight more advanced because of the column-family concept).
>>>
>>> I think it makes sense to open a DISCUSS thread about this approach,
>> since
>>> Schema Inference is in itself a very nice distinguishing feature I think.
>>> I'd like to invite anyone to share their ideas here, so that this is
>> maybe
>>> a place where we can make MetaModel shine.
>>>
>>> Cheers,
>>> Kasper
>>>
>>> [1] https://reviews.apache.org/r/23228/
>>
Re: [DISCUSS] JSON DataContext and a 'Schema inference' aspect in the
'core' module
Posted by Kasper Sørensen <i....@gmail.com>.
Well almost. It would be a Map<String,Object>, since the value-type is
unknown. For instance, if the document had nested objects or arrays, or
even a number-value, it would not always be a string value. The metadata
would say that the column type is ColumnType.MAP
2014-07-08 12:56 GMT+02:00 Ankit Kumar <ak...@gmail.com>:
> Hi Kasper,
>
> Quick question - Will the map based interpretation of the JSON be a
> Map<String,String> type and would we get this automatically in that type
> when reading the column from the row.
>
> If that's the idea then I would vote for it [+1] as we can already benefit
> from this.
>
> If I interpret it incorrect then do please enlighten me.
>
> Regards
> Ankit
>
>
> On Thu, Jul 3, 2014 at 12:26 PM, Kasper Sørensen <
> i.am.kasper.sorensen@gmail.com> wrote:
>
> > Hi guys,
> >
> > Maybe you saw that I posted a review request [1] yesterday for
> METAMODEL-38
> > (a JSON based module for MetaModel).
> >
> > I was building this JSON module and trying to do it in a way where the
> user
> > could configure how the logical schema would look like. In some cases you
> > would want MetaModel to infer the schema based on a sample of documents
> in
> > the source, and in other cases you might want MetaModel to just treat the
> > source as a 1-column table with a MAP data type. There's probably also
> > other strategies.
> >
> > That part I felt was also very relevant for many other "schemaless"
> > datastores, such as MongoDB, CouchDB, HBase etc. So I put there
> interfaces
> > and a few standard implementations of it into the core module, and
> applied
> > it to the JSON module. If this idea is accepted, I would like to also add
> > it to MongoDB and CouchDB modules (those are a natural fit) and maybe
> also
> > HBase (slight more advanced because of the column-family concept).
> >
> > I think it makes sense to open a DISCUSS thread about this approach,
> since
> > Schema Inference is in itself a very nice distinguishing feature I think.
> > I'd like to invite anyone to share their ideas here, so that this is
> maybe
> > a place where we can make MetaModel shine.
> >
> > Cheers,
> > Kasper
> >
> > [1] https://reviews.apache.org/r/23228/
> >
>
Re: [DISCUSS] JSON DataContext and a 'Schema inference' aspect in the
'core' module
Posted by Ankit Kumar <ak...@gmail.com>.
Hi Kasper,
Quick question - Will the map based interpretation of the JSON be a
Map<String,String> type and would we get this automatically in that type
when reading the column from the row.
If that's the idea then I would vote for it [+1] as we can already benefit
from this.
If I interpret it incorrect then do please enlighten me.
Regards
Ankit
On Thu, Jul 3, 2014 at 12:26 PM, Kasper Sørensen <
i.am.kasper.sorensen@gmail.com> wrote:
> Hi guys,
>
> Maybe you saw that I posted a review request [1] yesterday for METAMODEL-38
> (a JSON based module for MetaModel).
>
> I was building this JSON module and trying to do it in a way where the user
> could configure how the logical schema would look like. In some cases you
> would want MetaModel to infer the schema based on a sample of documents in
> the source, and in other cases you might want MetaModel to just treat the
> source as a 1-column table with a MAP data type. There's probably also
> other strategies.
>
> That part I felt was also very relevant for many other "schemaless"
> datastores, such as MongoDB, CouchDB, HBase etc. So I put there interfaces
> and a few standard implementations of it into the core module, and applied
> it to the JSON module. If this idea is accepted, I would like to also add
> it to MongoDB and CouchDB modules (those are a natural fit) and maybe also
> HBase (slight more advanced because of the column-family concept).
>
> I think it makes sense to open a DISCUSS thread about this approach, since
> Schema Inference is in itself a very nice distinguishing feature I think.
> I'd like to invite anyone to share their ideas here, so that this is maybe
> a place where we can make MetaModel shine.
>
> Cheers,
> Kasper
>
> [1] https://reviews.apache.org/r/23228/
>
Re: [DISCUSS] JSON DataContext and a 'Schema inference' aspect in the
'core' module
Posted by Kasper Sørensen <i....@gmail.com>.
A good topic for a discussion-within-the-discussion here is (column) type
inference. Right now we check what class the objects in a document has. But
we could maybe go a bit further to check especially for Strings what type
of content there is. If the String always represent a number for instance,
we could potentially do something there to set the ColumnType as NUMBER and
always convert the string automatically. Not sure what you think of that?
The danger there is of course that it could break if just a single document
is then not living up to that convention.
2014-07-03 12:26 GMT+02:00 Kasper Sørensen <i....@gmail.com>:
> Hi guys,
>
> Maybe you saw that I posted a review request [1] yesterday
> for METAMODEL-38 (a JSON based module for MetaModel).
>
> I was building this JSON module and trying to do it in a way where the
> user could configure how the logical schema would look like. In some cases
> you would want MetaModel to infer the schema based on a sample of documents
> in the source, and in other cases you might want MetaModel to just treat
> the source as a 1-column table with a MAP data type. There's probably also
> other strategies.
>
> That part I felt was also very relevant for many other "schemaless"
> datastores, such as MongoDB, CouchDB, HBase etc. So I put there interfaces
> and a few standard implementations of it into the core module, and applied
> it to the JSON module. If this idea is accepted, I would like to also add
> it to MongoDB and CouchDB modules (those are a natural fit) and maybe also
> HBase (slight more advanced because of the column-family concept).
>
> I think it makes sense to open a DISCUSS thread about this approach, since
> Schema Inference is in itself a very nice distinguishing feature I think.
> I'd like to invite anyone to share their ideas here, so that this is maybe
> a place where we can make MetaModel shine.
>
> Cheers,
> Kasper
>
> [1] https://reviews.apache.org/r/23228/
>
Re: [DISCUSS] JSON DataContext and a 'Schema inference' aspect in the
'core' module
Posted by Kasper Sørensen <i....@gmail.com>.
I feel like adding the new JSON module, but would love to get some more
opinions on the approach taken here ... What do people think?
2014-07-03 19:01 GMT+02:00 Henry Saputra <he...@gmail.com>:
> Thanks Kasper, need some time to review and try it out =)
>
> On Thu, Jul 3, 2014 at 3:26 AM, Kasper Sørensen
> <i....@gmail.com> wrote:
> > Hi guys,
> >
> > Maybe you saw that I posted a review request [1] yesterday for
> METAMODEL-38
> > (a JSON based module for MetaModel).
> >
> > I was building this JSON module and trying to do it in a way where the
> user
> > could configure how the logical schema would look like. In some cases you
> > would want MetaModel to infer the schema based on a sample of documents
> in
> > the source, and in other cases you might want MetaModel to just treat the
> > source as a 1-column table with a MAP data type. There's probably also
> > other strategies.
> >
> > That part I felt was also very relevant for many other "schemaless"
> > datastores, such as MongoDB, CouchDB, HBase etc. So I put there
> interfaces
> > and a few standard implementations of it into the core module, and
> applied
> > it to the JSON module. If this idea is accepted, I would like to also add
> > it to MongoDB and CouchDB modules (those are a natural fit) and maybe
> also
> > HBase (slight more advanced because of the column-family concept).
> >
> > I think it makes sense to open a DISCUSS thread about this approach,
> since
> > Schema Inference is in itself a very nice distinguishing feature I think.
> > I'd like to invite anyone to share their ideas here, so that this is
> maybe
> > a place where we can make MetaModel shine.
> >
> > Cheers,
> > Kasper
> >
> > [1] https://reviews.apache.org/r/23228/
>
Re: [DISCUSS] JSON DataContext and a 'Schema inference' aspect in the
'core' module
Posted by Henry Saputra <he...@gmail.com>.
Thanks Kasper, need some time to review and try it out =)
On Thu, Jul 3, 2014 at 3:26 AM, Kasper Sørensen
<i....@gmail.com> wrote:
> Hi guys,
>
> Maybe you saw that I posted a review request [1] yesterday for METAMODEL-38
> (a JSON based module for MetaModel).
>
> I was building this JSON module and trying to do it in a way where the user
> could configure how the logical schema would look like. In some cases you
> would want MetaModel to infer the schema based on a sample of documents in
> the source, and in other cases you might want MetaModel to just treat the
> source as a 1-column table with a MAP data type. There's probably also
> other strategies.
>
> That part I felt was also very relevant for many other "schemaless"
> datastores, such as MongoDB, CouchDB, HBase etc. So I put there interfaces
> and a few standard implementations of it into the core module, and applied
> it to the JSON module. If this idea is accepted, I would like to also add
> it to MongoDB and CouchDB modules (those are a natural fit) and maybe also
> HBase (slight more advanced because of the column-family concept).
>
> I think it makes sense to open a DISCUSS thread about this approach, since
> Schema Inference is in itself a very nice distinguishing feature I think.
> I'd like to invite anyone to share their ideas here, so that this is maybe
> a place where we can make MetaModel shine.
>
> Cheers,
> Kasper
>
> [1] https://reviews.apache.org/r/23228/
Re: [DISCUSS] JSON DataContext and a 'Schema inference' aspect in the
'core' module
Posted by Kasper Sørensen <i....@gmail.com>.
Oh yes there should be. Maybe we should make a separate JIRA for that kind
of stuff. I think also the schema builder concept, and the built-in
inferential implementations, deserve a wiki page to explain them to users.
2014-07-10 18:56 GMT+02:00 Henry Saputra <he...@gmail.com>:
> The only missing part I guess the documentation.
>
> Will there be update to the website to reflect the new JSON module?
>
> - Henry
>
> On Thu, Jul 3, 2014 at 3:26 AM, Kasper Sørensen
> <i....@gmail.com> wrote:
> > Hi guys,
> >
> > Maybe you saw that I posted a review request [1] yesterday for
> METAMODEL-38
> > (a JSON based module for MetaModel).
> >
> > I was building this JSON module and trying to do it in a way where the
> user
> > could configure how the logical schema would look like. In some cases you
> > would want MetaModel to infer the schema based on a sample of documents
> in
> > the source, and in other cases you might want MetaModel to just treat the
> > source as a 1-column table with a MAP data type. There's probably also
> > other strategies.
> >
> > That part I felt was also very relevant for many other "schemaless"
> > datastores, such as MongoDB, CouchDB, HBase etc. So I put there
> interfaces
> > and a few standard implementations of it into the core module, and
> applied
> > it to the JSON module. If this idea is accepted, I would like to also add
> > it to MongoDB and CouchDB modules (those are a natural fit) and maybe
> also
> > HBase (slight more advanced because of the column-family concept).
> >
> > I think it makes sense to open a DISCUSS thread about this approach,
> since
> > Schema Inference is in itself a very nice distinguishing feature I think.
> > I'd like to invite anyone to share their ideas here, so that this is
> maybe
> > a place where we can make MetaModel shine.
> >
> > Cheers,
> > Kasper
> >
> > [1] https://reviews.apache.org/r/23228/
>
Re: [DISCUSS] JSON DataContext and a 'Schema inference' aspect in the
'core' module
Posted by Henry Saputra <he...@gmail.com>.
The only missing part I guess the documentation.
Will there be update to the website to reflect the new JSON module?
- Henry
On Thu, Jul 3, 2014 at 3:26 AM, Kasper Sørensen
<i....@gmail.com> wrote:
> Hi guys,
>
> Maybe you saw that I posted a review request [1] yesterday for METAMODEL-38
> (a JSON based module for MetaModel).
>
> I was building this JSON module and trying to do it in a way where the user
> could configure how the logical schema would look like. In some cases you
> would want MetaModel to infer the schema based on a sample of documents in
> the source, and in other cases you might want MetaModel to just treat the
> source as a 1-column table with a MAP data type. There's probably also
> other strategies.
>
> That part I felt was also very relevant for many other "schemaless"
> datastores, such as MongoDB, CouchDB, HBase etc. So I put there interfaces
> and a few standard implementations of it into the core module, and applied
> it to the JSON module. If this idea is accepted, I would like to also add
> it to MongoDB and CouchDB modules (those are a natural fit) and maybe also
> HBase (slight more advanced because of the column-family concept).
>
> I think it makes sense to open a DISCUSS thread about this approach, since
> Schema Inference is in itself a very nice distinguishing feature I think.
> I'd like to invite anyone to share their ideas here, so that this is maybe
> a place where we can make MetaModel shine.
>
> Cheers,
> Kasper
>
> [1] https://reviews.apache.org/r/23228/