You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metamodel.apache.org by Kasper Sørensen <i....@gmail.com> on 2015/07/10 12:51:05 UTC
Adding more metadata (for instance about scale or other features) in MetaModel
Hi all,
All the time I see more and more need for us to add metadata to our
MetaModel based connectors. That could be for instance metadata about scale
(nominal, ordinal etc.) so that we can automate some stats collection etc.
or it could be more "meaning" oriented features to describe e.g. "This is a
first name" or "This is a city" or "These two fields (first and last name)
are together defining a name of a person".
We have such mechanisms in our application levels many places, but not at
the core framework of MetaModel and that's a pity because it makes it
harder for us to share.
So I'm thinking of adding such a layer to the metadata of MetaModel. But
one thing that's difficult is then about representing that metadata in some
"metadata store" which isn't necesarily the same as the data source itself.
It could be an XML file or it could be a complete metadata database. And I
think that this metadata would be mutable by the integrator of MetaModel
because it is rarely fully revealed by the data source itself.
What do you think? Nice feature or?`
Kasper
Re: Adding more metadata (for instance about scale or other features)
in MetaModel
Posted by Kasper Sørensen <i....@gmail.com>.
An additional note/idea that just came to mind: Some features like these
might be detectable and some might even be intrinsically available in the
data source. For instance in Salesforce.com many of the fields are standard
fields as well as the tables being standard tables. That means that for
this data source we could add features coming from the DataContext itself.
My suggestion would be to add an optional interface for DataContexts that
could expose such informations.
As for detection I need to think a bit more, or maybe some of you have
ideas. Maybe something a la the schema inference stuff we have where a
detector object gets a preview of the data in order to try and determine
metadata.
2015-07-15 19:41 GMT+02:00 Kasper Sørensen <i....@gmail.com>:
> That's absolutely a good idea. One thing is describing the JSON structure,
> which I guess would not be different (although more elaborate than average)
> than other features. But the moment you mention this then I start thinking
> about also query "into" the JSON. Or any other map structure. Also many of
> our NoSQL-like modules have more advanced data structures - for instance
> MongoDB, CouchDB, ElasticSearch and so on.
>
> Querying into those data structures could be considered a separate feature
> though. I mean - that could be useful regardless of having metadata about
> the data structure. Even today I would love to somehow query with a
> function or something like that where I could provide a kind of path (XPath
> like or so) into the nested structure. Ankit, if this is also what you
> would like, then I suggest starting a separate mail thread on it (unless
> you feel it is tied very closely to this metadata thing).
>
> 2015-07-15 18:55 GMT+02:00 Ankit Kumar <ak...@gmail.com>:
>
>> Hi Kasper,
>>
>> Sounds like a nice idea.
>>
>> Would this also allow to define a complex metadata on a single column. For
>> e.g. if the content of the column is a JSON object, would it be possible
>> to
>> define what is the structure of the JSON, what specific fields within the
>> JSON structure have what behavioural aspect (metadata as well)? May be an
>> extreme case for later, but I can already see the benefit of this feature.
>>
>> Regards
>> Ankit
>>
>> On Wed, Jul 15, 2015 at 4:58 PM, Kasper Sørensen <
>> i.am.kasper.sorensen@gmail.com> wrote:
>>
>> > Glad that you like the idea :-) And yes, I also would like to involve as
>> > many as possible in building it.
>> >
>> > My high level design ideas:
>> >
>> > We would leave the existing DataContext API as-is to the extent
>> possible. I
>> > would see this as a service that is connected to the DataContext but
>> since
>> > the "new" metadata and the existing metadata might differ (as Alberto
>> > mentioned before) it is also not bound directly to the schema, table,
>> > column etc. Rather you should be able to resolve things using either
>> > Schema, Table and Column objects, or simply by paths (strings).
>> >
>> > Here's some example method calls and class names that I could imagine. I
>> > haven't thought long about the naming so it's just a "idea generating"
>> > draft. I imagine the metadata "features" to be pluggable kinda like
>> > annotations on java constructs:
>> >
>> > DataContext dc = ...
>> > MetadataService svc = ...
>> >
>> > Table table = dc.getDefaultSchema().getTable(0);
>> >
>> > List<ColumnGroupMetadata> groups = svc.getColumnGroupMetadata(table);
>> > boolean isAddressGroup =
>> > groups.get(0).hasFeatureOfType(AddressColumnGroupFeature.class);
>> > AddressColumnGroupFeature addressFeature =
>> > groups.get(0).getFeatureOfType(AddressColumnGroupFeature.class);
>> > ColumnMetadata countryColumnMetadata =
>> addressFeature.getCountryColumn();
>> > List<ColumnMetadata> addressLineColumnMetadataList =
>> > addressFeature.getAddressLineColumns();
>> >
>> > Column column = table.getColumn(0);
>> >
>> > ColumnMetadata cm = svc.getColumnMetadata(column);
>> > List<ColumnFeature> features = cm.getFeatures();
>> > boolean isNominal = cm.hasFeatureOfType(NominalColumnFeature.class);
>> > NominalColumnFeature ncf =
>> cm.getFeatureOfType(NominalColumnFeature.class);
>> >
>> >
>> > BR,
>> > Kasper
>> >
>> >
>> > 2015-07-15 13:07 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
>> >
>> > > That's pretty cool. Sorry but I was not taking into account the
>> powerful
>> > > queries that you might bring into the equation with this stuff. It all
>> > > makes sense now.
>> > >
>> > > It would definitely be a big feature that might be divided into
>> smaller
>> > > tasks. If we go ahead with this stuff I really would like to help with
>> > the
>> > > implementation.
>> > >
>> > > Kind regards,
>> > >
>> > > Alberto
>> > >
>> > > 2015-07-15 12:41 GMT+02:00 Kasper Sørensen <
>> > i.am.kasper.sorensen@gmail.com
>> > > >:
>> > >
>> > > > That's absolutely true. It's not that I want to stop discovering
>> what
>> > we
>> > > > can, but I was more thinking of also adding a mechanism to plug your
>> > own
>> > > > metadata. I guess it's pretty rare that a database itself offers a
>> > > "domain
>> > > > oriented" metadata system where I could tell it that "this field is
>> a
>> > zip
>> > > > code, and together with field X, Y and Z it forms a single address".
>> > > >
>> > > > While from a querying perspective what I would love to archieve is
>> > that I
>> > > > could express something like the following:
>> > > >
>> > > > "Query all the addresses in the database".
>> > > > "Do a SUM, AVG, MAX and MIN on all the ordinal-scale numbers in the
>> > > > database".
>> > > >
>> > > > It will also help a lot in generating templates for data
>> integration.
>> > If
>> > > I
>> > > > am trying to move data from one table to another then I can probably
>> > do a
>> > > > lot of automatic mapping based on the metadata. Same goes for
>> > reporting I
>> > > > guess and stuff like that.
>> > > >
>> > > > Best regards,
>> > > > Kasper
>> > > >
>> > > > 2015-07-15 12:06 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
>> > > >
>> > > > > Ok, so you are not thinking of discovering more metadata
>> > "on-the-fly",
>> > > > your
>> > > > > approach is statically define metadata for the datasource and load
>> > and
>> > > > > mix-in it with the existing metadata right?
>> > > > >
>> > > > > IMHO with this approach we will add a strong dependency between
>> the
>> > > data
>> > > > > itself and the "external" metadata, correct me if I'm wrong or not
>> > > fully
>> > > > > understand your proposal but if the datastore changes (one column
>> is
>> > > > > deleted or a new column is added) the metadata will get obsolete.
>> > > > >
>> > > > > Kind regards,
>> > > > >
>> > > > > Alberto
>> > > > >
>> > > > > 2015-07-13 19:26 GMT+02:00 Kasper Sørensen <
>> > > > i.am.kasper.sorensen@gmail.com
>> > > > > >:
>> > > > >
>> > > > > > I was thinking of having something like pluggable annotations or
>> > > > features
>> > > > > > that could be added to tables, columns or groups of columns.
>> Maybe
>> > > also
>> > > > > to
>> > > > > > other entities. But since a lot of this is not available as a
>> thing
>> > > > that
>> > > > > > can be explored in the datastore itself I guess it would need
>> to be
>> > > > > stored
>> > > > > > externally.
>> > > > > >
>> > > > > > Examples of features that I could imagine:
>> > > > > >
>> > > > > > Data type features: NominalScale, OrdinalScale, ...
>> > > > > > Data conversion features: IntegerAsString, DateAsString,
>> > > > TimestampAsLong
>> > > > > > ...
>> > > > > > Domain features: FirstName, LastName, AddressLine, AddressCity,
>> > > > > > AddressCountry, DateYear, DateMonth ...
>> > > > > > And groupings of columns also in domain like fatures: Name
>> > (composed
>> > > of
>> > > > > > e.g. first and last name), Address (composed of multiple address
>> > > > fields),
>> > > > > > Date (composed of year, month etc.)
>> > > > > >
>> > > > > > I would like to store them so that I can save and load them,
>> saving
>> > > the
>> > > > > > developer for the work of restoring all the metadata again and
>> > again.
>> > > > Is
>> > > > > > that not sensible?
>> > > > > >
>> > > > > > Best regards,
>> > > > > > Kasper
>> > > > > >
>> > > > > >
>> > > > > > 2015-07-13 9:55 GMT+02:00 Alberto Rodriguez <ardlema@gmail.com
>> >:
>> > > > > >
>> > > > > > > Hi all,
>> > > > > > >
>> > > > > > > We are also facing similar issues so I completely agree with
>> this
>> > > > > > feature.
>> > > > > > > In fact, we added recently in our service layer a new field
>> for
>> > our
>> > > > > > > metadata called "format", we needed this field to specify
>> > different
>> > > > > > format
>> > > > > > > types for the dates returned by our datastores.
>> > > > > > >
>> > > > > > > However, I'm not really sure how to implement this feature...
>> I
>> > > guess
>> > > > > we
>> > > > > > > should keep getting the metadata "core" from our datasources
>> but
>> > > what
>> > > > > > about
>> > > > > > > the new metadata??:
>> > > > > > >
>> > > > > > > - How to fill it out? Will the integrator of MM provide
>> > > functions
>> > > > to
>> > > > > > > define when an element is going to be "x" and when is going
>> to be
>> > > > "y"?
>> > > > > > > (thinking here of providing lambda functions)
>> > > > > > > - Do we really need a metadata store?
>> > > > > > >
>> > > > > > > Kind regards,
>> > > > > > >
>> > > > > > >
>> > > > > > > 2015-07-10 12:51 GMT+02:00 Kasper Sørensen <
>> > > > > > i.am.kasper.sorensen@gmail.com
>> > > > > > > >:
>> > > > > > >
>> > > > > > > > Hi all,
>> > > > > > > >
>> > > > > > > > All the time I see more and more need for us to add
>> metadata to
>> > > our
>> > > > > > > > MetaModel based connectors. That could be for instance
>> metadata
>> > > > about
>> > > > > > > scale
>> > > > > > > > (nominal, ordinal etc.) so that we can automate some stats
>> > > > collection
>> > > > > > > etc.
>> > > > > > > > or it could be more "meaning" oriented features to describe
>> > e.g.
>> > > > > "This
>> > > > > > > is a
>> > > > > > > > first name" or "This is a city" or "These two fields (first
>> and
>> > > > last
>> > > > > > > name)
>> > > > > > > > are together defining a name of a person".
>> > > > > > > >
>> > > > > > > > We have such mechanisms in our application levels many
>> places,
>> > > but
>> > > > > not
>> > > > > > at
>> > > > > > > > the core framework of MetaModel and that's a pity because it
>> > > makes
>> > > > it
>> > > > > > > > harder for us to share.
>> > > > > > > >
>> > > > > > > > So I'm thinking of adding such a layer to the metadata of
>> > > > MetaModel.
>> > > > > > But
>> > > > > > > > one thing that's difficult is then about representing that
>> > > metadata
>> > > > > in
>> > > > > > > some
>> > > > > > > > "metadata store" which isn't necesarily the same as the data
>> > > source
>> > > > > > > itself.
>> > > > > > > > It could be an XML file or it could be a complete metadata
>> > > > database.
>> > > > > > And
>> > > > > > > I
>> > > > > > > > think that this metadata would be mutable by the integrator
>> of
>> > > > > > MetaModel
>> > > > > > > > because it is rarely fully revealed by the data source
>> itself.
>> > > > > > > >
>> > > > > > > > What do you think? Nice feature or?`
>> > > > > > > >
>> > > > > > > > Kasper
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Adding more metadata (for instance about scale or other features)
in MetaModel
Posted by Kasper Sørensen <i....@gmail.com>.
That's absolutely a good idea. One thing is describing the JSON structure,
which I guess would not be different (although more elaborate than average)
than other features. But the moment you mention this then I start thinking
about also query "into" the JSON. Or any other map structure. Also many of
our NoSQL-like modules have more advanced data structures - for instance
MongoDB, CouchDB, ElasticSearch and so on.
Querying into those data structures could be considered a separate feature
though. I mean - that could be useful regardless of having metadata about
the data structure. Even today I would love to somehow query with a
function or something like that where I could provide a kind of path (XPath
like or so) into the nested structure. Ankit, if this is also what you
would like, then I suggest starting a separate mail thread on it (unless
you feel it is tied very closely to this metadata thing).
2015-07-15 18:55 GMT+02:00 Ankit Kumar <ak...@gmail.com>:
> Hi Kasper,
>
> Sounds like a nice idea.
>
> Would this also allow to define a complex metadata on a single column. For
> e.g. if the content of the column is a JSON object, would it be possible to
> define what is the structure of the JSON, what specific fields within the
> JSON structure have what behavioural aspect (metadata as well)? May be an
> extreme case for later, but I can already see the benefit of this feature.
>
> Regards
> Ankit
>
> On Wed, Jul 15, 2015 at 4:58 PM, Kasper Sørensen <
> i.am.kasper.sorensen@gmail.com> wrote:
>
> > Glad that you like the idea :-) And yes, I also would like to involve as
> > many as possible in building it.
> >
> > My high level design ideas:
> >
> > We would leave the existing DataContext API as-is to the extent
> possible. I
> > would see this as a service that is connected to the DataContext but
> since
> > the "new" metadata and the existing metadata might differ (as Alberto
> > mentioned before) it is also not bound directly to the schema, table,
> > column etc. Rather you should be able to resolve things using either
> > Schema, Table and Column objects, or simply by paths (strings).
> >
> > Here's some example method calls and class names that I could imagine. I
> > haven't thought long about the naming so it's just a "idea generating"
> > draft. I imagine the metadata "features" to be pluggable kinda like
> > annotations on java constructs:
> >
> > DataContext dc = ...
> > MetadataService svc = ...
> >
> > Table table = dc.getDefaultSchema().getTable(0);
> >
> > List<ColumnGroupMetadata> groups = svc.getColumnGroupMetadata(table);
> > boolean isAddressGroup =
> > groups.get(0).hasFeatureOfType(AddressColumnGroupFeature.class);
> > AddressColumnGroupFeature addressFeature =
> > groups.get(0).getFeatureOfType(AddressColumnGroupFeature.class);
> > ColumnMetadata countryColumnMetadata = addressFeature.getCountryColumn();
> > List<ColumnMetadata> addressLineColumnMetadataList =
> > addressFeature.getAddressLineColumns();
> >
> > Column column = table.getColumn(0);
> >
> > ColumnMetadata cm = svc.getColumnMetadata(column);
> > List<ColumnFeature> features = cm.getFeatures();
> > boolean isNominal = cm.hasFeatureOfType(NominalColumnFeature.class);
> > NominalColumnFeature ncf =
> cm.getFeatureOfType(NominalColumnFeature.class);
> >
> >
> > BR,
> > Kasper
> >
> >
> > 2015-07-15 13:07 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
> >
> > > That's pretty cool. Sorry but I was not taking into account the
> powerful
> > > queries that you might bring into the equation with this stuff. It all
> > > makes sense now.
> > >
> > > It would definitely be a big feature that might be divided into smaller
> > > tasks. If we go ahead with this stuff I really would like to help with
> > the
> > > implementation.
> > >
> > > Kind regards,
> > >
> > > Alberto
> > >
> > > 2015-07-15 12:41 GMT+02:00 Kasper Sørensen <
> > i.am.kasper.sorensen@gmail.com
> > > >:
> > >
> > > > That's absolutely true. It's not that I want to stop discovering what
> > we
> > > > can, but I was more thinking of also adding a mechanism to plug your
> > own
> > > > metadata. I guess it's pretty rare that a database itself offers a
> > > "domain
> > > > oriented" metadata system where I could tell it that "this field is a
> > zip
> > > > code, and together with field X, Y and Z it forms a single address".
> > > >
> > > > While from a querying perspective what I would love to archieve is
> > that I
> > > > could express something like the following:
> > > >
> > > > "Query all the addresses in the database".
> > > > "Do a SUM, AVG, MAX and MIN on all the ordinal-scale numbers in the
> > > > database".
> > > >
> > > > It will also help a lot in generating templates for data integration.
> > If
> > > I
> > > > am trying to move data from one table to another then I can probably
> > do a
> > > > lot of automatic mapping based on the metadata. Same goes for
> > reporting I
> > > > guess and stuff like that.
> > > >
> > > > Best regards,
> > > > Kasper
> > > >
> > > > 2015-07-15 12:06 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
> > > >
> > > > > Ok, so you are not thinking of discovering more metadata
> > "on-the-fly",
> > > > your
> > > > > approach is statically define metadata for the datasource and load
> > and
> > > > > mix-in it with the existing metadata right?
> > > > >
> > > > > IMHO with this approach we will add a strong dependency between the
> > > data
> > > > > itself and the "external" metadata, correct me if I'm wrong or not
> > > fully
> > > > > understand your proposal but if the datastore changes (one column
> is
> > > > > deleted or a new column is added) the metadata will get obsolete.
> > > > >
> > > > > Kind regards,
> > > > >
> > > > > Alberto
> > > > >
> > > > > 2015-07-13 19:26 GMT+02:00 Kasper Sørensen <
> > > > i.am.kasper.sorensen@gmail.com
> > > > > >:
> > > > >
> > > > > > I was thinking of having something like pluggable annotations or
> > > > features
> > > > > > that could be added to tables, columns or groups of columns.
> Maybe
> > > also
> > > > > to
> > > > > > other entities. But since a lot of this is not available as a
> thing
> > > > that
> > > > > > can be explored in the datastore itself I guess it would need to
> be
> > > > > stored
> > > > > > externally.
> > > > > >
> > > > > > Examples of features that I could imagine:
> > > > > >
> > > > > > Data type features: NominalScale, OrdinalScale, ...
> > > > > > Data conversion features: IntegerAsString, DateAsString,
> > > > TimestampAsLong
> > > > > > ...
> > > > > > Domain features: FirstName, LastName, AddressLine, AddressCity,
> > > > > > AddressCountry, DateYear, DateMonth ...
> > > > > > And groupings of columns also in domain like fatures: Name
> > (composed
> > > of
> > > > > > e.g. first and last name), Address (composed of multiple address
> > > > fields),
> > > > > > Date (composed of year, month etc.)
> > > > > >
> > > > > > I would like to store them so that I can save and load them,
> saving
> > > the
> > > > > > developer for the work of restoring all the metadata again and
> > again.
> > > > Is
> > > > > > that not sensible?
> > > > > >
> > > > > > Best regards,
> > > > > > Kasper
> > > > > >
> > > > > >
> > > > > > 2015-07-13 9:55 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We are also facing similar issues so I completely agree with
> this
> > > > > > feature.
> > > > > > > In fact, we added recently in our service layer a new field for
> > our
> > > > > > > metadata called "format", we needed this field to specify
> > different
> > > > > > format
> > > > > > > types for the dates returned by our datastores.
> > > > > > >
> > > > > > > However, I'm not really sure how to implement this feature... I
> > > guess
> > > > > we
> > > > > > > should keep getting the metadata "core" from our datasources
> but
> > > what
> > > > > > about
> > > > > > > the new metadata??:
> > > > > > >
> > > > > > > - How to fill it out? Will the integrator of MM provide
> > > functions
> > > > to
> > > > > > > define when an element is going to be "x" and when is going to
> be
> > > > "y"?
> > > > > > > (thinking here of providing lambda functions)
> > > > > > > - Do we really need a metadata store?
> > > > > > >
> > > > > > > Kind regards,
> > > > > > >
> > > > > > >
> > > > > > > 2015-07-10 12:51 GMT+02:00 Kasper Sørensen <
> > > > > > i.am.kasper.sorensen@gmail.com
> > > > > > > >:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > All the time I see more and more need for us to add metadata
> to
> > > our
> > > > > > > > MetaModel based connectors. That could be for instance
> metadata
> > > > about
> > > > > > > scale
> > > > > > > > (nominal, ordinal etc.) so that we can automate some stats
> > > > collection
> > > > > > > etc.
> > > > > > > > or it could be more "meaning" oriented features to describe
> > e.g.
> > > > > "This
> > > > > > > is a
> > > > > > > > first name" or "This is a city" or "These two fields (first
> and
> > > > last
> > > > > > > name)
> > > > > > > > are together defining a name of a person".
> > > > > > > >
> > > > > > > > We have such mechanisms in our application levels many
> places,
> > > but
> > > > > not
> > > > > > at
> > > > > > > > the core framework of MetaModel and that's a pity because it
> > > makes
> > > > it
> > > > > > > > harder for us to share.
> > > > > > > >
> > > > > > > > So I'm thinking of adding such a layer to the metadata of
> > > > MetaModel.
> > > > > > But
> > > > > > > > one thing that's difficult is then about representing that
> > > metadata
> > > > > in
> > > > > > > some
> > > > > > > > "metadata store" which isn't necesarily the same as the data
> > > source
> > > > > > > itself.
> > > > > > > > It could be an XML file or it could be a complete metadata
> > > > database.
> > > > > > And
> > > > > > > I
> > > > > > > > think that this metadata would be mutable by the integrator
> of
> > > > > > MetaModel
> > > > > > > > because it is rarely fully revealed by the data source
> itself.
> > > > > > > >
> > > > > > > > What do you think? Nice feature or?`
> > > > > > > >
> > > > > > > > Kasper
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Adding more metadata (for instance about scale or other features)
in MetaModel
Posted by Ankit Kumar <ak...@gmail.com>.
Hi Kasper,
Sounds like a nice idea.
Would this also allow to define a complex metadata on a single column. For
e.g. if the content of the column is a JSON object, would it be possible to
define what is the structure of the JSON, what specific fields within the
JSON structure have what behavioural aspect (metadata as well)? May be an
extreme case for later, but I can already see the benefit of this feature.
Regards
Ankit
On Wed, Jul 15, 2015 at 4:58 PM, Kasper Sørensen <
i.am.kasper.sorensen@gmail.com> wrote:
> Glad that you like the idea :-) And yes, I also would like to involve as
> many as possible in building it.
>
> My high level design ideas:
>
> We would leave the existing DataContext API as-is to the extent possible. I
> would see this as a service that is connected to the DataContext but since
> the "new" metadata and the existing metadata might differ (as Alberto
> mentioned before) it is also not bound directly to the schema, table,
> column etc. Rather you should be able to resolve things using either
> Schema, Table and Column objects, or simply by paths (strings).
>
> Here's some example method calls and class names that I could imagine. I
> haven't thought long about the naming so it's just a "idea generating"
> draft. I imagine the metadata "features" to be pluggable kinda like
> annotations on java constructs:
>
> DataContext dc = ...
> MetadataService svc = ...
>
> Table table = dc.getDefaultSchema().getTable(0);
>
> List<ColumnGroupMetadata> groups = svc.getColumnGroupMetadata(table);
> boolean isAddressGroup =
> groups.get(0).hasFeatureOfType(AddressColumnGroupFeature.class);
> AddressColumnGroupFeature addressFeature =
> groups.get(0).getFeatureOfType(AddressColumnGroupFeature.class);
> ColumnMetadata countryColumnMetadata = addressFeature.getCountryColumn();
> List<ColumnMetadata> addressLineColumnMetadataList =
> addressFeature.getAddressLineColumns();
>
> Column column = table.getColumn(0);
>
> ColumnMetadata cm = svc.getColumnMetadata(column);
> List<ColumnFeature> features = cm.getFeatures();
> boolean isNominal = cm.hasFeatureOfType(NominalColumnFeature.class);
> NominalColumnFeature ncf = cm.getFeatureOfType(NominalColumnFeature.class);
>
>
> BR,
> Kasper
>
>
> 2015-07-15 13:07 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
>
> > That's pretty cool. Sorry but I was not taking into account the powerful
> > queries that you might bring into the equation with this stuff. It all
> > makes sense now.
> >
> > It would definitely be a big feature that might be divided into smaller
> > tasks. If we go ahead with this stuff I really would like to help with
> the
> > implementation.
> >
> > Kind regards,
> >
> > Alberto
> >
> > 2015-07-15 12:41 GMT+02:00 Kasper Sørensen <
> i.am.kasper.sorensen@gmail.com
> > >:
> >
> > > That's absolutely true. It's not that I want to stop discovering what
> we
> > > can, but I was more thinking of also adding a mechanism to plug your
> own
> > > metadata. I guess it's pretty rare that a database itself offers a
> > "domain
> > > oriented" metadata system where I could tell it that "this field is a
> zip
> > > code, and together with field X, Y and Z it forms a single address".
> > >
> > > While from a querying perspective what I would love to archieve is
> that I
> > > could express something like the following:
> > >
> > > "Query all the addresses in the database".
> > > "Do a SUM, AVG, MAX and MIN on all the ordinal-scale numbers in the
> > > database".
> > >
> > > It will also help a lot in generating templates for data integration.
> If
> > I
> > > am trying to move data from one table to another then I can probably
> do a
> > > lot of automatic mapping based on the metadata. Same goes for
> reporting I
> > > guess and stuff like that.
> > >
> > > Best regards,
> > > Kasper
> > >
> > > 2015-07-15 12:06 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
> > >
> > > > Ok, so you are not thinking of discovering more metadata
> "on-the-fly",
> > > your
> > > > approach is statically define metadata for the datasource and load
> and
> > > > mix-in it with the existing metadata right?
> > > >
> > > > IMHO with this approach we will add a strong dependency between the
> > data
> > > > itself and the "external" metadata, correct me if I'm wrong or not
> > fully
> > > > understand your proposal but if the datastore changes (one column is
> > > > deleted or a new column is added) the metadata will get obsolete.
> > > >
> > > > Kind regards,
> > > >
> > > > Alberto
> > > >
> > > > 2015-07-13 19:26 GMT+02:00 Kasper Sørensen <
> > > i.am.kasper.sorensen@gmail.com
> > > > >:
> > > >
> > > > > I was thinking of having something like pluggable annotations or
> > > features
> > > > > that could be added to tables, columns or groups of columns. Maybe
> > also
> > > > to
> > > > > other entities. But since a lot of this is not available as a thing
> > > that
> > > > > can be explored in the datastore itself I guess it would need to be
> > > > stored
> > > > > externally.
> > > > >
> > > > > Examples of features that I could imagine:
> > > > >
> > > > > Data type features: NominalScale, OrdinalScale, ...
> > > > > Data conversion features: IntegerAsString, DateAsString,
> > > TimestampAsLong
> > > > > ...
> > > > > Domain features: FirstName, LastName, AddressLine, AddressCity,
> > > > > AddressCountry, DateYear, DateMonth ...
> > > > > And groupings of columns also in domain like fatures: Name
> (composed
> > of
> > > > > e.g. first and last name), Address (composed of multiple address
> > > fields),
> > > > > Date (composed of year, month etc.)
> > > > >
> > > > > I would like to store them so that I can save and load them, saving
> > the
> > > > > developer for the work of restoring all the metadata again and
> again.
> > > Is
> > > > > that not sensible?
> > > > >
> > > > > Best regards,
> > > > > Kasper
> > > > >
> > > > >
> > > > > 2015-07-13 9:55 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We are also facing similar issues so I completely agree with this
> > > > > feature.
> > > > > > In fact, we added recently in our service layer a new field for
> our
> > > > > > metadata called "format", we needed this field to specify
> different
> > > > > format
> > > > > > types for the dates returned by our datastores.
> > > > > >
> > > > > > However, I'm not really sure how to implement this feature... I
> > guess
> > > > we
> > > > > > should keep getting the metadata "core" from our datasources but
> > what
> > > > > about
> > > > > > the new metadata??:
> > > > > >
> > > > > > - How to fill it out? Will the integrator of MM provide
> > functions
> > > to
> > > > > > define when an element is going to be "x" and when is going to be
> > > "y"?
> > > > > > (thinking here of providing lambda functions)
> > > > > > - Do we really need a metadata store?
> > > > > >
> > > > > > Kind regards,
> > > > > >
> > > > > >
> > > > > > 2015-07-10 12:51 GMT+02:00 Kasper Sørensen <
> > > > > i.am.kasper.sorensen@gmail.com
> > > > > > >:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > All the time I see more and more need for us to add metadata to
> > our
> > > > > > > MetaModel based connectors. That could be for instance metadata
> > > about
> > > > > > scale
> > > > > > > (nominal, ordinal etc.) so that we can automate some stats
> > > collection
> > > > > > etc.
> > > > > > > or it could be more "meaning" oriented features to describe
> e.g.
> > > > "This
> > > > > > is a
> > > > > > > first name" or "This is a city" or "These two fields (first and
> > > last
> > > > > > name)
> > > > > > > are together defining a name of a person".
> > > > > > >
> > > > > > > We have such mechanisms in our application levels many places,
> > but
> > > > not
> > > > > at
> > > > > > > the core framework of MetaModel and that's a pity because it
> > makes
> > > it
> > > > > > > harder for us to share.
> > > > > > >
> > > > > > > So I'm thinking of adding such a layer to the metadata of
> > > MetaModel.
> > > > > But
> > > > > > > one thing that's difficult is then about representing that
> > metadata
> > > > in
> > > > > > some
> > > > > > > "metadata store" which isn't necesarily the same as the data
> > source
> > > > > > itself.
> > > > > > > It could be an XML file or it could be a complete metadata
> > > database.
> > > > > And
> > > > > > I
> > > > > > > think that this metadata would be mutable by the integrator of
> > > > > MetaModel
> > > > > > > because it is rarely fully revealed by the data source itself.
> > > > > > >
> > > > > > > What do you think? Nice feature or?`
> > > > > > >
> > > > > > > Kasper
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Adding more metadata (for instance about scale or other features)
in MetaModel
Posted by Kasper Sørensen <i....@gmail.com>.
Glad that you like the idea :-) And yes, I also would like to involve as
many as possible in building it.
My high level design ideas:
We would leave the existing DataContext API as-is to the extent possible. I
would see this as a service that is connected to the DataContext but since
the "new" metadata and the existing metadata might differ (as Alberto
mentioned before) it is also not bound directly to the schema, table,
column etc. Rather you should be able to resolve things using either
Schema, Table and Column objects, or simply by paths (strings).
Here's some example method calls and class names that I could imagine. I
haven't thought long about the naming so it's just a "idea generating"
draft. I imagine the metadata "features" to be pluggable kinda like
annotations on java constructs:
DataContext dc = ...
MetadataService svc = ...
Table table = dc.getDefaultSchema().getTable(0);
List<ColumnGroupMetadata> groups = svc.getColumnGroupMetadata(table);
boolean isAddressGroup =
groups.get(0).hasFeatureOfType(AddressColumnGroupFeature.class);
AddressColumnGroupFeature addressFeature =
groups.get(0).getFeatureOfType(AddressColumnGroupFeature.class);
ColumnMetadata countryColumnMetadata = addressFeature.getCountryColumn();
List<ColumnMetadata> addressLineColumnMetadataList =
addressFeature.getAddressLineColumns();
Column column = table.getColumn(0);
ColumnMetadata cm = svc.getColumnMetadata(column);
List<ColumnFeature> features = cm.getFeatures();
boolean isNominal = cm.hasFeatureOfType(NominalColumnFeature.class);
NominalColumnFeature ncf = cm.getFeatureOfType(NominalColumnFeature.class);
BR,
Kasper
2015-07-15 13:07 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
> That's pretty cool. Sorry but I was not taking into account the powerful
> queries that you might bring into the equation with this stuff. It all
> makes sense now.
>
> It would definitely be a big feature that might be divided into smaller
> tasks. If we go ahead with this stuff I really would like to help with the
> implementation.
>
> Kind regards,
>
> Alberto
>
> 2015-07-15 12:41 GMT+02:00 Kasper Sørensen <i.am.kasper.sorensen@gmail.com
> >:
>
> > That's absolutely true. It's not that I want to stop discovering what we
> > can, but I was more thinking of also adding a mechanism to plug your own
> > metadata. I guess it's pretty rare that a database itself offers a
> "domain
> > oriented" metadata system where I could tell it that "this field is a zip
> > code, and together with field X, Y and Z it forms a single address".
> >
> > While from a querying perspective what I would love to archieve is that I
> > could express something like the following:
> >
> > "Query all the addresses in the database".
> > "Do a SUM, AVG, MAX and MIN on all the ordinal-scale numbers in the
> > database".
> >
> > It will also help a lot in generating templates for data integration. If
> I
> > am trying to move data from one table to another then I can probably do a
> > lot of automatic mapping based on the metadata. Same goes for reporting I
> > guess and stuff like that.
> >
> > Best regards,
> > Kasper
> >
> > 2015-07-15 12:06 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
> >
> > > Ok, so you are not thinking of discovering more metadata "on-the-fly",
> > your
> > > approach is statically define metadata for the datasource and load and
> > > mix-in it with the existing metadata right?
> > >
> > > IMHO with this approach we will add a strong dependency between the
> data
> > > itself and the "external" metadata, correct me if I'm wrong or not
> fully
> > > understand your proposal but if the datastore changes (one column is
> > > deleted or a new column is added) the metadata will get obsolete.
> > >
> > > Kind regards,
> > >
> > > Alberto
> > >
> > > 2015-07-13 19:26 GMT+02:00 Kasper Sørensen <
> > i.am.kasper.sorensen@gmail.com
> > > >:
> > >
> > > > I was thinking of having something like pluggable annotations or
> > features
> > > > that could be added to tables, columns or groups of columns. Maybe
> also
> > > to
> > > > other entities. But since a lot of this is not available as a thing
> > that
> > > > can be explored in the datastore itself I guess it would need to be
> > > stored
> > > > externally.
> > > >
> > > > Examples of features that I could imagine:
> > > >
> > > > Data type features: NominalScale, OrdinalScale, ...
> > > > Data conversion features: IntegerAsString, DateAsString,
> > TimestampAsLong
> > > > ...
> > > > Domain features: FirstName, LastName, AddressLine, AddressCity,
> > > > AddressCountry, DateYear, DateMonth ...
> > > > And groupings of columns also in domain like fatures: Name (composed
> of
> > > > e.g. first and last name), Address (composed of multiple address
> > fields),
> > > > Date (composed of year, month etc.)
> > > >
> > > > I would like to store them so that I can save and load them, saving
> the
> > > > developer for the work of restoring all the metadata again and again.
> > Is
> > > > that not sensible?
> > > >
> > > > Best regards,
> > > > Kasper
> > > >
> > > >
> > > > 2015-07-13 9:55 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
> > > >
> > > > > Hi all,
> > > > >
> > > > > We are also facing similar issues so I completely agree with this
> > > > feature.
> > > > > In fact, we added recently in our service layer a new field for our
> > > > > metadata called "format", we needed this field to specify different
> > > > format
> > > > > types for the dates returned by our datastores.
> > > > >
> > > > > However, I'm not really sure how to implement this feature... I
> guess
> > > we
> > > > > should keep getting the metadata "core" from our datasources but
> what
> > > > about
> > > > > the new metadata??:
> > > > >
> > > > > - How to fill it out? Will the integrator of MM provide
> functions
> > to
> > > > > define when an element is going to be "x" and when is going to be
> > "y"?
> > > > > (thinking here of providing lambda functions)
> > > > > - Do we really need a metadata store?
> > > > >
> > > > > Kind regards,
> > > > >
> > > > >
> > > > > 2015-07-10 12:51 GMT+02:00 Kasper Sørensen <
> > > > i.am.kasper.sorensen@gmail.com
> > > > > >:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > All the time I see more and more need for us to add metadata to
> our
> > > > > > MetaModel based connectors. That could be for instance metadata
> > about
> > > > > scale
> > > > > > (nominal, ordinal etc.) so that we can automate some stats
> > collection
> > > > > etc.
> > > > > > or it could be more "meaning" oriented features to describe e.g.
> > > "This
> > > > > is a
> > > > > > first name" or "This is a city" or "These two fields (first and
> > last
> > > > > name)
> > > > > > are together defining a name of a person".
> > > > > >
> > > > > > We have such mechanisms in our application levels many places,
> but
> > > not
> > > > at
> > > > > > the core framework of MetaModel and that's a pity because it
> makes
> > it
> > > > > > harder for us to share.
> > > > > >
> > > > > > So I'm thinking of adding such a layer to the metadata of
> > MetaModel.
> > > > But
> > > > > > one thing that's difficult is then about representing that
> metadata
> > > in
> > > > > some
> > > > > > "metadata store" which isn't necesarily the same as the data
> source
> > > > > itself.
> > > > > > It could be an XML file or it could be a complete metadata
> > database.
> > > > And
> > > > > I
> > > > > > think that this metadata would be mutable by the integrator of
> > > > MetaModel
> > > > > > because it is rarely fully revealed by the data source itself.
> > > > > >
> > > > > > What do you think? Nice feature or?`
> > > > > >
> > > > > > Kasper
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Adding more metadata (for instance about scale or other features)
in MetaModel
Posted by Alberto Rodriguez <ar...@gmail.com>.
That's pretty cool. Sorry but I was not taking into account the powerful
queries that you might bring into the equation with this stuff. It all
makes sense now.
It would definitely be a big feature that might be divided into smaller
tasks. If we go ahead with this stuff I really would like to help with the
implementation.
Kind regards,
Alberto
2015-07-15 12:41 GMT+02:00 Kasper Sørensen <i....@gmail.com>:
> That's absolutely true. It's not that I want to stop discovering what we
> can, but I was more thinking of also adding a mechanism to plug your own
> metadata. I guess it's pretty rare that a database itself offers a "domain
> oriented" metadata system where I could tell it that "this field is a zip
> code, and together with field X, Y and Z it forms a single address".
>
> While from a querying perspective what I would love to archieve is that I
> could express something like the following:
>
> "Query all the addresses in the database".
> "Do a SUM, AVG, MAX and MIN on all the ordinal-scale numbers in the
> database".
>
> It will also help a lot in generating templates for data integration. If I
> am trying to move data from one table to another then I can probably do a
> lot of automatic mapping based on the metadata. Same goes for reporting I
> guess and stuff like that.
>
> Best regards,
> Kasper
>
> 2015-07-15 12:06 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
>
> > Ok, so you are not thinking of discovering more metadata "on-the-fly",
> your
> > approach is statically define metadata for the datasource and load and
> > mix-in it with the existing metadata right?
> >
> > IMHO with this approach we will add a strong dependency between the data
> > itself and the "external" metadata, correct me if I'm wrong or not fully
> > understand your proposal but if the datastore changes (one column is
> > deleted or a new column is added) the metadata will get obsolete.
> >
> > Kind regards,
> >
> > Alberto
> >
> > 2015-07-13 19:26 GMT+02:00 Kasper Sørensen <
> i.am.kasper.sorensen@gmail.com
> > >:
> >
> > > I was thinking of having something like pluggable annotations or
> features
> > > that could be added to tables, columns or groups of columns. Maybe also
> > to
> > > other entities. But since a lot of this is not available as a thing
> that
> > > can be explored in the datastore itself I guess it would need to be
> > stored
> > > externally.
> > >
> > > Examples of features that I could imagine:
> > >
> > > Data type features: NominalScale, OrdinalScale, ...
> > > Data conversion features: IntegerAsString, DateAsString,
> TimestampAsLong
> > > ...
> > > Domain features: FirstName, LastName, AddressLine, AddressCity,
> > > AddressCountry, DateYear, DateMonth ...
> > > And groupings of columns also in domain like fatures: Name (composed of
> > > e.g. first and last name), Address (composed of multiple address
> fields),
> > > Date (composed of year, month etc.)
> > >
> > > I would like to store them so that I can save and load them, saving the
> > > developer for the work of restoring all the metadata again and again.
> Is
> > > that not sensible?
> > >
> > > Best regards,
> > > Kasper
> > >
> > >
> > > 2015-07-13 9:55 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
> > >
> > > > Hi all,
> > > >
> > > > We are also facing similar issues so I completely agree with this
> > > feature.
> > > > In fact, we added recently in our service layer a new field for our
> > > > metadata called "format", we needed this field to specify different
> > > format
> > > > types for the dates returned by our datastores.
> > > >
> > > > However, I'm not really sure how to implement this feature... I guess
> > we
> > > > should keep getting the metadata "core" from our datasources but what
> > > about
> > > > the new metadata??:
> > > >
> > > > - How to fill it out? Will the integrator of MM provide functions
> to
> > > > define when an element is going to be "x" and when is going to be
> "y"?
> > > > (thinking here of providing lambda functions)
> > > > - Do we really need a metadata store?
> > > >
> > > > Kind regards,
> > > >
> > > >
> > > > 2015-07-10 12:51 GMT+02:00 Kasper Sørensen <
> > > i.am.kasper.sorensen@gmail.com
> > > > >:
> > > >
> > > > > Hi all,
> > > > >
> > > > > All the time I see more and more need for us to add metadata to our
> > > > > MetaModel based connectors. That could be for instance metadata
> about
> > > > scale
> > > > > (nominal, ordinal etc.) so that we can automate some stats
> collection
> > > > etc.
> > > > > or it could be more "meaning" oriented features to describe e.g.
> > "This
> > > > is a
> > > > > first name" or "This is a city" or "These two fields (first and
> last
> > > > name)
> > > > > are together defining a name of a person".
> > > > >
> > > > > We have such mechanisms in our application levels many places, but
> > not
> > > at
> > > > > the core framework of MetaModel and that's a pity because it makes
> it
> > > > > harder for us to share.
> > > > >
> > > > > So I'm thinking of adding such a layer to the metadata of
> MetaModel.
> > > But
> > > > > one thing that's difficult is then about representing that metadata
> > in
> > > > some
> > > > > "metadata store" which isn't necesarily the same as the data source
> > > > itself.
> > > > > It could be an XML file or it could be a complete metadata
> database.
> > > And
> > > > I
> > > > > think that this metadata would be mutable by the integrator of
> > > MetaModel
> > > > > because it is rarely fully revealed by the data source itself.
> > > > >
> > > > > What do you think? Nice feature or?`
> > > > >
> > > > > Kasper
> > > > >
> > > >
> > >
> >
>
Re: Adding more metadata (for instance about scale or other features)
in MetaModel
Posted by Kasper Sørensen <i....@gmail.com>.
That's absolutely true. It's not that I want to stop discovering what we
can, but I was more thinking of also adding a mechanism to plug your own
metadata. I guess it's pretty rare that a database itself offers a "domain
oriented" metadata system where I could tell it that "this field is a zip
code, and together with field X, Y and Z it forms a single address".
While from a querying perspective what I would love to archieve is that I
could express something like the following:
"Query all the addresses in the database".
"Do a SUM, AVG, MAX and MIN on all the ordinal-scale numbers in the
database".
It will also help a lot in generating templates for data integration. If I
am trying to move data from one table to another then I can probably do a
lot of automatic mapping based on the metadata. Same goes for reporting I
guess and stuff like that.
Best regards,
Kasper
2015-07-15 12:06 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
> Ok, so you are not thinking of discovering more metadata "on-the-fly", your
> approach is statically define metadata for the datasource and load and
> mix-in it with the existing metadata right?
>
> IMHO with this approach we will add a strong dependency between the data
> itself and the "external" metadata, correct me if I'm wrong or not fully
> understand your proposal but if the datastore changes (one column is
> deleted or a new column is added) the metadata will get obsolete.
>
> Kind regards,
>
> Alberto
>
> 2015-07-13 19:26 GMT+02:00 Kasper Sørensen <i.am.kasper.sorensen@gmail.com
> >:
>
> > I was thinking of having something like pluggable annotations or features
> > that could be added to tables, columns or groups of columns. Maybe also
> to
> > other entities. But since a lot of this is not available as a thing that
> > can be explored in the datastore itself I guess it would need to be
> stored
> > externally.
> >
> > Examples of features that I could imagine:
> >
> > Data type features: NominalScale, OrdinalScale, ...
> > Data conversion features: IntegerAsString, DateAsString, TimestampAsLong
> > ...
> > Domain features: FirstName, LastName, AddressLine, AddressCity,
> > AddressCountry, DateYear, DateMonth ...
> > And groupings of columns also in domain like fatures: Name (composed of
> > e.g. first and last name), Address (composed of multiple address fields),
> > Date (composed of year, month etc.)
> >
> > I would like to store them so that I can save and load them, saving the
> > developer for the work of restoring all the metadata again and again. Is
> > that not sensible?
> >
> > Best regards,
> > Kasper
> >
> >
> > 2015-07-13 9:55 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
> >
> > > Hi all,
> > >
> > > We are also facing similar issues so I completely agree with this
> > feature.
> > > In fact, we added recently in our service layer a new field for our
> > > metadata called "format", we needed this field to specify different
> > format
> > > types for the dates returned by our datastores.
> > >
> > > However, I'm not really sure how to implement this feature... I guess
> we
> > > should keep getting the metadata "core" from our datasources but what
> > about
> > > the new metadata??:
> > >
> > > - How to fill it out? Will the integrator of MM provide functions to
> > > define when an element is going to be "x" and when is going to be "y"?
> > > (thinking here of providing lambda functions)
> > > - Do we really need a metadata store?
> > >
> > > Kind regards,
> > >
> > >
> > > 2015-07-10 12:51 GMT+02:00 Kasper Sørensen <
> > i.am.kasper.sorensen@gmail.com
> > > >:
> > >
> > > > Hi all,
> > > >
> > > > All the time I see more and more need for us to add metadata to our
> > > > MetaModel based connectors. That could be for instance metadata about
> > > scale
> > > > (nominal, ordinal etc.) so that we can automate some stats collection
> > > etc.
> > > > or it could be more "meaning" oriented features to describe e.g.
> "This
> > > is a
> > > > first name" or "This is a city" or "These two fields (first and last
> > > name)
> > > > are together defining a name of a person".
> > > >
> > > > We have such mechanisms in our application levels many places, but
> not
> > at
> > > > the core framework of MetaModel and that's a pity because it makes it
> > > > harder for us to share.
> > > >
> > > > So I'm thinking of adding such a layer to the metadata of MetaModel.
> > But
> > > > one thing that's difficult is then about representing that metadata
> in
> > > some
> > > > "metadata store" which isn't necesarily the same as the data source
> > > itself.
> > > > It could be an XML file or it could be a complete metadata database.
> > And
> > > I
> > > > think that this metadata would be mutable by the integrator of
> > MetaModel
> > > > because it is rarely fully revealed by the data source itself.
> > > >
> > > > What do you think? Nice feature or?`
> > > >
> > > > Kasper
> > > >
> > >
> >
>
Re: Adding more metadata (for instance about scale or other features)
in MetaModel
Posted by Alberto Rodriguez <ar...@gmail.com>.
Ok, so you are not thinking of discovering more metadata "on-the-fly", your
approach is statically define metadata for the datasource and load and
mix-in it with the existing metadata right?
IMHO with this approach we will add a strong dependency between the data
itself and the "external" metadata, correct me if I'm wrong or not fully
understand your proposal but if the datastore changes (one column is
deleted or a new column is added) the metadata will get obsolete.
Kind regards,
Alberto
2015-07-13 19:26 GMT+02:00 Kasper Sørensen <i....@gmail.com>:
> I was thinking of having something like pluggable annotations or features
> that could be added to tables, columns or groups of columns. Maybe also to
> other entities. But since a lot of this is not available as a thing that
> can be explored in the datastore itself I guess it would need to be stored
> externally.
>
> Examples of features that I could imagine:
>
> Data type features: NominalScale, OrdinalScale, ...
> Data conversion features: IntegerAsString, DateAsString, TimestampAsLong
> ...
> Domain features: FirstName, LastName, AddressLine, AddressCity,
> AddressCountry, DateYear, DateMonth ...
> And groupings of columns also in domain like fatures: Name (composed of
> e.g. first and last name), Address (composed of multiple address fields),
> Date (composed of year, month etc.)
>
> I would like to store them so that I can save and load them, saving the
> developer for the work of restoring all the metadata again and again. Is
> that not sensible?
>
> Best regards,
> Kasper
>
>
> 2015-07-13 9:55 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
>
> > Hi all,
> >
> > We are also facing similar issues so I completely agree with this
> feature.
> > In fact, we added recently in our service layer a new field for our
> > metadata called "format", we needed this field to specify different
> format
> > types for the dates returned by our datastores.
> >
> > However, I'm not really sure how to implement this feature... I guess we
> > should keep getting the metadata "core" from our datasources but what
> about
> > the new metadata??:
> >
> > - How to fill it out? Will the integrator of MM provide functions to
> > define when an element is going to be "x" and when is going to be "y"?
> > (thinking here of providing lambda functions)
> > - Do we really need a metadata store?
> >
> > Kind regards,
> >
> >
> > 2015-07-10 12:51 GMT+02:00 Kasper Sørensen <
> i.am.kasper.sorensen@gmail.com
> > >:
> >
> > > Hi all,
> > >
> > > All the time I see more and more need for us to add metadata to our
> > > MetaModel based connectors. That could be for instance metadata about
> > scale
> > > (nominal, ordinal etc.) so that we can automate some stats collection
> > etc.
> > > or it could be more "meaning" oriented features to describe e.g. "This
> > is a
> > > first name" or "This is a city" or "These two fields (first and last
> > name)
> > > are together defining a name of a person".
> > >
> > > We have such mechanisms in our application levels many places, but not
> at
> > > the core framework of MetaModel and that's a pity because it makes it
> > > harder for us to share.
> > >
> > > So I'm thinking of adding such a layer to the metadata of MetaModel.
> But
> > > one thing that's difficult is then about representing that metadata in
> > some
> > > "metadata store" which isn't necesarily the same as the data source
> > itself.
> > > It could be an XML file or it could be a complete metadata database.
> And
> > I
> > > think that this metadata would be mutable by the integrator of
> MetaModel
> > > because it is rarely fully revealed by the data source itself.
> > >
> > > What do you think? Nice feature or?`
> > >
> > > Kasper
> > >
> >
>
Re: Adding more metadata (for instance about scale or other features)
in MetaModel
Posted by Kasper Sørensen <i....@gmail.com>.
I was thinking of having something like pluggable annotations or features
that could be added to tables, columns or groups of columns. Maybe also to
other entities. But since a lot of this is not available as a thing that
can be explored in the datastore itself I guess it would need to be stored
externally.
Examples of features that I could imagine:
Data type features: NominalScale, OrdinalScale, ...
Data conversion features: IntegerAsString, DateAsString, TimestampAsLong ...
Domain features: FirstName, LastName, AddressLine, AddressCity,
AddressCountry, DateYear, DateMonth ...
And groupings of columns also in domain like fatures: Name (composed of
e.g. first and last name), Address (composed of multiple address fields),
Date (composed of year, month etc.)
I would like to store them so that I can save and load them, saving the
developer for the work of restoring all the metadata again and again. Is
that not sensible?
Best regards,
Kasper
2015-07-13 9:55 GMT+02:00 Alberto Rodriguez <ar...@gmail.com>:
> Hi all,
>
> We are also facing similar issues so I completely agree with this feature.
> In fact, we added recently in our service layer a new field for our
> metadata called "format", we needed this field to specify different format
> types for the dates returned by our datastores.
>
> However, I'm not really sure how to implement this feature... I guess we
> should keep getting the metadata "core" from our datasources but what about
> the new metadata??:
>
> - How to fill it out? Will the integrator of MM provide functions to
> define when an element is going to be "x" and when is going to be "y"?
> (thinking here of providing lambda functions)
> - Do we really need a metadata store?
>
> Kind regards,
>
>
> 2015-07-10 12:51 GMT+02:00 Kasper Sørensen <i.am.kasper.sorensen@gmail.com
> >:
>
> > Hi all,
> >
> > All the time I see more and more need for us to add metadata to our
> > MetaModel based connectors. That could be for instance metadata about
> scale
> > (nominal, ordinal etc.) so that we can automate some stats collection
> etc.
> > or it could be more "meaning" oriented features to describe e.g. "This
> is a
> > first name" or "This is a city" or "These two fields (first and last
> name)
> > are together defining a name of a person".
> >
> > We have such mechanisms in our application levels many places, but not at
> > the core framework of MetaModel and that's a pity because it makes it
> > harder for us to share.
> >
> > So I'm thinking of adding such a layer to the metadata of MetaModel. But
> > one thing that's difficult is then about representing that metadata in
> some
> > "metadata store" which isn't necesarily the same as the data source
> itself.
> > It could be an XML file or it could be a complete metadata database. And
> I
> > think that this metadata would be mutable by the integrator of MetaModel
> > because it is rarely fully revealed by the data source itself.
> >
> > What do you think? Nice feature or?`
> >
> > Kasper
> >
>
Re: Adding more metadata (for instance about scale or other features)
in MetaModel
Posted by Alberto Rodriguez <ar...@gmail.com>.
Hi all,
We are also facing similar issues so I completely agree with this feature.
In fact, we added recently in our service layer a new field for our
metadata called "format", we needed this field to specify different format
types for the dates returned by our datastores.
However, I'm not really sure how to implement this feature... I guess we
should keep getting the metadata "core" from our datasources but what about
the new metadata??:
- How to fill it out? Will the integrator of MM provide functions to
define when an element is going to be "x" and when is going to be "y"?
(thinking here of providing lambda functions)
- Do we really need a metadata store?
Kind regards,
2015-07-10 12:51 GMT+02:00 Kasper Sørensen <i....@gmail.com>:
> Hi all,
>
> All the time I see more and more need for us to add metadata to our
> MetaModel based connectors. That could be for instance metadata about scale
> (nominal, ordinal etc.) so that we can automate some stats collection etc.
> or it could be more "meaning" oriented features to describe e.g. "This is a
> first name" or "This is a city" or "These two fields (first and last name)
> are together defining a name of a person".
>
> We have such mechanisms in our application levels many places, but not at
> the core framework of MetaModel and that's a pity because it makes it
> harder for us to share.
>
> So I'm thinking of adding such a layer to the metadata of MetaModel. But
> one thing that's difficult is then about representing that metadata in some
> "metadata store" which isn't necesarily the same as the data source itself.
> It could be an XML file or it could be a complete metadata database. And I
> think that this metadata would be mutable by the integrator of MetaModel
> because it is rarely fully revealed by the data source itself.
>
> What do you think? Nice feature or?`
>
> Kasper
>