You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@asterixdb.apache.org by Riyafa Abdul Hameed <ri...@apache.org> on 2017/06/21 09:26:04 UTC
Parse GeoJSON data into a record in AsterixDB
Hi,
I would like to parse the following or any GeoJSON type[1] to a record in
AsterixDB:
{
"type":"Feature",
"geometry":{
"type":"Point",
"coordinates":[
-118.40,
33.93
]
},
"properties":{
"code":"LAX",
"elevation":38
}
}
The value of properties is optional and is a variable that is it can be any
type of object. What is the most suitable datatype to use to represent
properties in this case?
Is something like the following possible?
CREATE TYPE GeometryType AS {
type: string,
geometry: GeometryType,
properties: object
};
I came up with the above because there's a derived type called objects[2]
in AsterixDB. The above doesn't work because of the following reasons:
- type appears to be keyword
- We can't use the defining type within the same type recursively (ie.
GeometryType within GeometryType)
- The type object cannot be resolved
Any suggestions on how a GeoJSON can be parsed into AsterixDB?
[1] https://tools.ietf.org/html/rfc7946
[2]
https://ci.apache.org/projects/asterixdb/datamodel.html#DerivedTypesObject
Thank you
Yours sincerely,
Riyafa
Re: Parse GeoJSON data into a record in AsterixDB
Posted by Till Westmann <ti...@apache.org>.
Glancing at RFC 7946 [1] it seems that arbitrary nesting is not allowed.
If I glanced correctly, there are
1) geometry objects and
2) feature objects that can contain a geometry and
3) feature collections that can contain multiple features.
So I believe that recursion should not be a problem.
However, I think that there are other constructs in the structure of a
geometry object that are not expressible in ADM types today - so I
think that we’ll need to work with (at least partially) open types.
Cheers,
Till
[1] https://tools.ietf.org/html/rfc7946
On 21 Jun 2017, at 15:33, Mike Carey wrote:
> One approach would be to be silent about properties - and then it
> could be there anyway - however, that wouldn't allow you to state the
> requirement (?) that it must be called properties and/or that it must
> be an object (not a scalar). That could work for now, perhaps? We
> need to have an "any record type" type name - we've noted a desire for
> that - unfortunately we don't have one I don't think. I believe the
> concept is there inside the code, in the type-related areas, but we
> don't have a keyword like name for it. (@Yingyi - comments?) And we
> do also have a restriction (at the type level) that precludes
> recursion (regular or mutual) in type definitions; we probably need to
> do something about that someday as well.
>
> In the meantime, these things could be handled (weakly) by documenting
> what's expected/allowed in this setting.
>
> Cheers,
>
> Mike
>
> PS - I wonder if JSON Schema has the expressiveness for this?
>
>
> On 6/21/17 2:26 AM, Riyafa Abdul Hameed wrote:
>> Hi,
>>
>> I would like to parse the following or any GeoJSON type[1] to a
>> record in AsterixDB:
>> {
>> "type":"Feature",
>> "geometry":{
>> "type":"Point",
>> "coordinates":[
>> -118.40,
>> 33.93
>> ]
>> },
>> "properties":{
>> "code":"LAX",
>> "elevation":38
>> }
>> }
>>
>> The value of properties is optional and is a variable that is it can
>> be any type of object. What is the most suitable datatype to use to
>> represent properties in this case?
>> Is something like the following possible?
>>
>> CREATE TYPE GeometryType AS {
>> type: string,
>> geometry: GeometryType,
>> properties: object
>> };
>>
>> I came up with the above because there's a derived type called
>> objects[2] in AsterixDB. The above doesn't work because of the
>> following reasons:
>>
>> * type appears to be keyword
>> * We can't use the defining type within the same type recursively
>> (ie. GeometryType within GeometryType)
>> * The type object cannot be resolved
>>
>> Any suggestions on how a GeoJSON can be parsed into AsterixDB?
>>
>> [1] https://tools.ietf.org/html/rfc7946
>> [2]
>> https://ci.apache.org/projects/asterixdb/datamodel.html#DerivedTypesObject
>>
>> Thank you
>> Yours sincerely,
>> Riyafa
>>
Re: Parse GeoJSON data into a record in AsterixDB
Posted by Michael Carey <mj...@ics.uci.edu>.
Cool!
On 6/22/17 7:58 AM, Riyafa Abdul Hameed wrote:
> Dear all,
>
> Thank you very much. I hadn't thought of an AnyObject type. Now I am
> able to parse GeoJSON using the following:
>
> DROP DATAVERSE GeoData IF EXISTS;
> CREATE DATAVERSE GeoData;
> USE GeoData;
> CREATE TYPE AnyObject AS {};
> CREATE TYPE FeatureType AS {
> id: UUID,
> `type`: string,
> `geometry`: AnyObject,
> properties: AnyObject
> };
> CREATE DATASET Geometries (FeatureType) PRIMARY KEY id autogenerated;
> INSERT INTO Geometries ([
> {
>
> "type":"Feature",
> "geometry":{
> "type":"Point",
> "coordinates":[
> -118.40,
> 33.93
> ]
> },
> "properties":{
> "code":"LAX",
> "elevation":38
> }
> }
> ]);
> SELECT * FROM Geometries;
>
> Thanks again.
> Riyafa
>
> On 22 June 2017 at 02:37, Ahmed Eldawy <eldawy@cs.ucr.edu
> <ma...@cs.ucr.edu>> wrote:
>
> Hi Riyafa,
>
> I think you should use the terms *feature* and *geometry* to avoid
> confusion like the following example.
>
> CREATE TYPE AnyObject AS {};
>
> CREATE TYPE *FeatureType* AS {
> `type`: string,
> the_geom: *geometry*,
> properties: AnyObject
> };
>
> The internal 'geometry' attribute is what holds the shape geometry
> while the outer FeatureType associates additional attributes and
> properties to that geometry.
>
> As we discussed in our last call, the first step is to parse
> GeoJSON as a regular JSON file and then parse the geometry using a
> UDF. In this case, you might have something like:
>
> CREATE TYPE *FeatureType* AS {
> `type`: string,
> the_geom: *AnyObject*,
> properties: AnyObject
> };
>
> This will allow you to parse the file without modifying the
> existing JSON parser. Then, you can define a UDF called
> "ParseGeoJSON" which takes as input the "the_geom" attribute and
> returns a parsed geometry attribute.
> The next step should avoid this additional step and should
> automatically detect and parse the geometry attribute directly
> from GeoJJSON.
>
>
> Thanks
> Ahmed
>
> On Wed, Jun 21, 2017 at 11:40 AM, Yingyi Bu <buyingyi@gmail.com
> <ma...@gmail.com>> wrote:
>
> >> type appears to be keyword
> `type` would make it valid.
>
> >> We can't use the defining type within the same type
> recursively (ie.
> GeometryType within GeometryType)
>
> We don't support recursive type definition.
>
> >> The type object cannot be resolved
>
> We don't have a builtin name for a completely open type, but
> you can
> define one.
>
>
> What you can do is:
>
> CREATE TYPE AnyObject AS {};
>
> CREATE TYPE GeometryType AS {
> `type`: string,
> geometry: SomeType;
> properties: AnyObject
> }
>
>
> Best,
> Yingyi
>
> On Wed, Jun 21, 2017 at 6:33 AM, Mike Carey <dtabass@gmail.com
> <ma...@gmail.com>> wrote:
>
> > One approach would be to be silent about properties - and
> then it could be
> > there anyway - however, that wouldn't allow you to state the
> requirement
> > (?) that it must be called properties and/or that it must be
> an object (not
> > a scalar). That could work for now, perhaps? We need to
> have an "any
> > record type" type name - we've noted a desire for that -
> unfortunately we
> > don't have one I don't think. I believe the concept is
> there inside the
> > code, in the type-related areas, but we don't have a keyword
> like name for
> > it. (@Yingyi - comments?) And we do also have a
> restriction (at the type
> > level) that precludes recursion (regular or mutual) in type
> definitions; we
> > probably need to do something about that someday as well.
> >
> > In the meantime, these things could be handled (weakly) by
> documenting
> > what's expected/allowed in this setting.
> >
> > Cheers,
> >
> > Mike
> >
> > PS - I wonder if JSON Schema has the expressiveness for this?
> >
> > On 6/21/17 2:26 AM, Riyafa Abdul Hameed wrote:
> >
> > Hi,
> >
> > I would like to parse the following or any GeoJSON type[1]
> to a record in
> > AsterixDB:
> > {
> > "type":"Feature",
> > "geometry":{
> > "type":"Point",
> > "coordinates":[
> > -118.40,
> > 33.93
> > ]
> > },
> > "properties":{
> > "code":"LAX",
> > "elevation":38
> > }
> > }
> >
> > The value of properties is optional and is a variable that
> is it can be
> > any type of object. What is the most suitable datatype to
> use to represent
> > properties in this case?
> > Is something like the following possible?
> >
> > CREATE TYPE GeometryType AS {
> > type: string,
> > geometry: GeometryType,
> > properties: object
> > };
> >
> > I came up with the above because there's a derived type
> called objects[2]
> > in AsterixDB. The above doesn't work because of the
> following reasons:
> >
> > - type appears to be keyword
> > - We can't use the defining type within the same type
> recursively (ie.
> > GeometryType within GeometryType)
> > - The type object cannot be resolved
> >
> > Any suggestions on how a GeoJSON can be parsed into AsterixDB?
> >
> > [1] https://tools.ietf.org/html/rfc7946
> <https://tools.ietf.org/html/rfc7946>
> > [2]
> https://ci.apache.org/projects/asterixdb/datamodel.html#Deri
> <https://ci.apache.org/projects/asterixdb/datamodel.html#Deri>
> > vedTypesObject
> >
> > Thank you
> > Yours sincerely,
> > Riyafa
> >
> >
> >
>
>
>
Re: Parse GeoJSON data into a record in AsterixDB
Posted by Riyafa Abdul Hameed <ri...@apache.org>.
Dear all,
Thank you very much. I hadn't thought of an AnyObject type. Now I am able
to parse GeoJSON using the following:
DROP DATAVERSE GeoData IF EXISTS;
CREATE DATAVERSE GeoData;
USE GeoData;
CREATE TYPE AnyObject AS {};
CREATE TYPE FeatureType AS {
id: UUID,
`type`: string,
`geometry`: AnyObject,
properties: AnyObject
};
CREATE DATASET Geometries (FeatureType) PRIMARY KEY id autogenerated;
INSERT INTO Geometries ([
{
"type":"Feature",
"geometry":{
"type":"Point",
"coordinates":[
-118.40,
33.93
]
},
"properties":{
"code":"LAX",
"elevation":38
}
}
]);
SELECT * FROM Geometries;
Thanks again.
Riyafa
On 22 June 2017 at 02:37, Ahmed Eldawy <el...@cs.ucr.edu> wrote:
> Hi Riyafa,
>
> I think you should use the terms *feature* and *geometry* to avoid
> confusion like the following example.
>
> CREATE TYPE AnyObject AS {};
>
> CREATE TYPE *FeatureType* AS {
> `type`: string,
> the_geom: *geometry*,
> properties: AnyObject
> };
>
> The internal 'geometry' attribute is what holds the shape geometry while
> the outer FeatureType associates additional attributes and properties to
> that geometry.
>
> As we discussed in our last call, the first step is to parse GeoJSON as a
> regular JSON file and then parse the geometry using a UDF. In this case,
> you might have something like:
>
> CREATE TYPE *FeatureType* AS {
> `type`: string,
> the_geom: *AnyObject*,
> properties: AnyObject
> };
>
> This will allow you to parse the file without modifying the existing JSON
> parser. Then, you can define a UDF called "ParseGeoJSON" which takes as
> input the "the_geom" attribute and returns a parsed geometry attribute.
> The next step should avoid this additional step and should automatically
> detect and parse the geometry attribute directly from GeoJJSON.
>
>
> Thanks
> Ahmed
>
> On Wed, Jun 21, 2017 at 11:40 AM, Yingyi Bu <bu...@gmail.com> wrote:
>
>> >> type appears to be keyword
>> `type` would make it valid.
>>
>> >> We can't use the defining type within the same type recursively (ie.
>> GeometryType within GeometryType)
>>
>> We don't support recursive type definition.
>>
>> >> The type object cannot be resolved
>>
>> We don't have a builtin name for a completely open type, but you can
>> define one.
>>
>>
>> What you can do is:
>>
>> CREATE TYPE AnyObject AS {};
>>
>> CREATE TYPE GeometryType AS {
>> `type`: string,
>> geometry: SomeType;
>> properties: AnyObject
>> }
>>
>>
>> Best,
>> Yingyi
>>
>> On Wed, Jun 21, 2017 at 6:33 AM, Mike Carey <dt...@gmail.com> wrote:
>>
>> > One approach would be to be silent about properties - and then it could
>> be
>> > there anyway - however, that wouldn't allow you to state the requirement
>> > (?) that it must be called properties and/or that it must be an object
>> (not
>> > a scalar). That could work for now, perhaps? We need to have an "any
>> > record type" type name - we've noted a desire for that - unfortunately
>> we
>> > don't have one I don't think. I believe the concept is there inside the
>> > code, in the type-related areas, but we don't have a keyword like name
>> for
>> > it. (@Yingyi - comments?) And we do also have a restriction (at the
>> type
>> > level) that precludes recursion (regular or mutual) in type
>> definitions; we
>> > probably need to do something about that someday as well.
>> >
>> > In the meantime, these things could be handled (weakly) by documenting
>> > what's expected/allowed in this setting.
>> >
>> > Cheers,
>> >
>> > Mike
>> >
>> > PS - I wonder if JSON Schema has the expressiveness for this?
>> >
>> > On 6/21/17 2:26 AM, Riyafa Abdul Hameed wrote:
>> >
>> > Hi,
>> >
>> > I would like to parse the following or any GeoJSON type[1] to a record
>> in
>> > AsterixDB:
>> > {
>> > "type":"Feature",
>> > "geometry":{
>> > "type":"Point",
>> > "coordinates":[
>> > -118.40,
>> > 33.93
>> > ]
>> > },
>> > "properties":{
>> > "code":"LAX",
>> > "elevation":38
>> > }
>> > }
>> >
>> > The value of properties is optional and is a variable that is it can be
>> > any type of object. What is the most suitable datatype to use to
>> represent
>> > properties in this case?
>> > Is something like the following possible?
>> >
>> > CREATE TYPE GeometryType AS {
>> > type: string,
>> > geometry: GeometryType,
>> > properties: object
>> > };
>> >
>> > I came up with the above because there's a derived type called
>> objects[2]
>> > in AsterixDB. The above doesn't work because of the following reasons:
>> >
>> > - type appears to be keyword
>> > - We can't use the defining type within the same type recursively
>> (ie.
>> > GeometryType within GeometryType)
>> > - The type object cannot be resolved
>> >
>> > Any suggestions on how a GeoJSON can be parsed into AsterixDB?
>> >
>> > [1] https://tools.ietf.org/html/rfc7946
>> > [2] https://ci.apache.org/projects/asterixdb/datamodel.html#Deri
>> > vedTypesObject
>> >
>> > Thank you
>> > Yours sincerely,
>> > Riyafa
>> >
>> >
>> >
>>
>
>
Re: Parse GeoJSON data into a record in AsterixDB
Posted by Riyafa Abdul Hameed <ri...@apache.org>.
Dear all,
Thank you very much. I hadn't thought of an AnyObject type. Now I am able
to parse GeoJSON using the following:
DROP DATAVERSE GeoData IF EXISTS;
CREATE DATAVERSE GeoData;
USE GeoData;
CREATE TYPE AnyObject AS {};
CREATE TYPE FeatureType AS {
id: UUID,
`type`: string,
`geometry`: AnyObject,
properties: AnyObject
};
CREATE DATASET Geometries (FeatureType) PRIMARY KEY id autogenerated;
INSERT INTO Geometries ([
{
"type":"Feature",
"geometry":{
"type":"Point",
"coordinates":[
-118.40,
33.93
]
},
"properties":{
"code":"LAX",
"elevation":38
}
}
]);
SELECT * FROM Geometries;
Thanks again.
Riyafa
On 22 June 2017 at 02:37, Ahmed Eldawy <el...@cs.ucr.edu> wrote:
> Hi Riyafa,
>
> I think you should use the terms *feature* and *geometry* to avoid
> confusion like the following example.
>
> CREATE TYPE AnyObject AS {};
>
> CREATE TYPE *FeatureType* AS {
> `type`: string,
> the_geom: *geometry*,
> properties: AnyObject
> };
>
> The internal 'geometry' attribute is what holds the shape geometry while
> the outer FeatureType associates additional attributes and properties to
> that geometry.
>
> As we discussed in our last call, the first step is to parse GeoJSON as a
> regular JSON file and then parse the geometry using a UDF. In this case,
> you might have something like:
>
> CREATE TYPE *FeatureType* AS {
> `type`: string,
> the_geom: *AnyObject*,
> properties: AnyObject
> };
>
> This will allow you to parse the file without modifying the existing JSON
> parser. Then, you can define a UDF called "ParseGeoJSON" which takes as
> input the "the_geom" attribute and returns a parsed geometry attribute.
> The next step should avoid this additional step and should automatically
> detect and parse the geometry attribute directly from GeoJJSON.
>
>
> Thanks
> Ahmed
>
> On Wed, Jun 21, 2017 at 11:40 AM, Yingyi Bu <bu...@gmail.com> wrote:
>
>> >> type appears to be keyword
>> `type` would make it valid.
>>
>> >> We can't use the defining type within the same type recursively (ie.
>> GeometryType within GeometryType)
>>
>> We don't support recursive type definition.
>>
>> >> The type object cannot be resolved
>>
>> We don't have a builtin name for a completely open type, but you can
>> define one.
>>
>>
>> What you can do is:
>>
>> CREATE TYPE AnyObject AS {};
>>
>> CREATE TYPE GeometryType AS {
>> `type`: string,
>> geometry: SomeType;
>> properties: AnyObject
>> }
>>
>>
>> Best,
>> Yingyi
>>
>> On Wed, Jun 21, 2017 at 6:33 AM, Mike Carey <dt...@gmail.com> wrote:
>>
>> > One approach would be to be silent about properties - and then it could
>> be
>> > there anyway - however, that wouldn't allow you to state the requirement
>> > (?) that it must be called properties and/or that it must be an object
>> (not
>> > a scalar). That could work for now, perhaps? We need to have an "any
>> > record type" type name - we've noted a desire for that - unfortunately
>> we
>> > don't have one I don't think. I believe the concept is there inside the
>> > code, in the type-related areas, but we don't have a keyword like name
>> for
>> > it. (@Yingyi - comments?) And we do also have a restriction (at the
>> type
>> > level) that precludes recursion (regular or mutual) in type
>> definitions; we
>> > probably need to do something about that someday as well.
>> >
>> > In the meantime, these things could be handled (weakly) by documenting
>> > what's expected/allowed in this setting.
>> >
>> > Cheers,
>> >
>> > Mike
>> >
>> > PS - I wonder if JSON Schema has the expressiveness for this?
>> >
>> > On 6/21/17 2:26 AM, Riyafa Abdul Hameed wrote:
>> >
>> > Hi,
>> >
>> > I would like to parse the following or any GeoJSON type[1] to a record
>> in
>> > AsterixDB:
>> > {
>> > "type":"Feature",
>> > "geometry":{
>> > "type":"Point",
>> > "coordinates":[
>> > -118.40,
>> > 33.93
>> > ]
>> > },
>> > "properties":{
>> > "code":"LAX",
>> > "elevation":38
>> > }
>> > }
>> >
>> > The value of properties is optional and is a variable that is it can be
>> > any type of object. What is the most suitable datatype to use to
>> represent
>> > properties in this case?
>> > Is something like the following possible?
>> >
>> > CREATE TYPE GeometryType AS {
>> > type: string,
>> > geometry: GeometryType,
>> > properties: object
>> > };
>> >
>> > I came up with the above because there's a derived type called
>> objects[2]
>> > in AsterixDB. The above doesn't work because of the following reasons:
>> >
>> > - type appears to be keyword
>> > - We can't use the defining type within the same type recursively
>> (ie.
>> > GeometryType within GeometryType)
>> > - The type object cannot be resolved
>> >
>> > Any suggestions on how a GeoJSON can be parsed into AsterixDB?
>> >
>> > [1] https://tools.ietf.org/html/rfc7946
>> > [2] https://ci.apache.org/projects/asterixdb/datamodel.html#Deri
>> > vedTypesObject
>> >
>> > Thank you
>> > Yours sincerely,
>> > Riyafa
>> >
>> >
>> >
>>
>
>
Re: Parse GeoJSON data into a record in AsterixDB
Posted by Ahmed Eldawy <el...@cs.ucr.edu>.
Hi Riyafa,
I think you should use the terms *feature* and *geometry* to avoid
confusion like the following example.
CREATE TYPE AnyObject AS {};
CREATE TYPE *FeatureType* AS {
`type`: string,
the_geom: *geometry*,
properties: AnyObject
};
The internal 'geometry' attribute is what holds the shape geometry while
the outer FeatureType associates additional attributes and properties to
that geometry.
As we discussed in our last call, the first step is to parse GeoJSON as a
regular JSON file and then parse the geometry using a UDF. In this case,
you might have something like:
CREATE TYPE *FeatureType* AS {
`type`: string,
the_geom: *AnyObject*,
properties: AnyObject
};
This will allow you to parse the file without modifying the existing JSON
parser. Then, you can define a UDF called "ParseGeoJSON" which takes as
input the "the_geom" attribute and returns a parsed geometry attribute.
The next step should avoid this additional step and should automatically
detect and parse the geometry attribute directly from GeoJJSON.
Thanks
Ahmed
On Wed, Jun 21, 2017 at 11:40 AM, Yingyi Bu <bu...@gmail.com> wrote:
> >> type appears to be keyword
> `type` would make it valid.
>
> >> We can't use the defining type within the same type recursively (ie.
> GeometryType within GeometryType)
>
> We don't support recursive type definition.
>
> >> The type object cannot be resolved
>
> We don't have a builtin name for a completely open type, but you can
> define one.
>
>
> What you can do is:
>
> CREATE TYPE AnyObject AS {};
>
> CREATE TYPE GeometryType AS {
> `type`: string,
> geometry: SomeType;
> properties: AnyObject
> }
>
>
> Best,
> Yingyi
>
> On Wed, Jun 21, 2017 at 6:33 AM, Mike Carey <dt...@gmail.com> wrote:
>
> > One approach would be to be silent about properties - and then it could
> be
> > there anyway - however, that wouldn't allow you to state the requirement
> > (?) that it must be called properties and/or that it must be an object
> (not
> > a scalar). That could work for now, perhaps? We need to have an "any
> > record type" type name - we've noted a desire for that - unfortunately we
> > don't have one I don't think. I believe the concept is there inside the
> > code, in the type-related areas, but we don't have a keyword like name
> for
> > it. (@Yingyi - comments?) And we do also have a restriction (at the
> type
> > level) that precludes recursion (regular or mutual) in type definitions;
> we
> > probably need to do something about that someday as well.
> >
> > In the meantime, these things could be handled (weakly) by documenting
> > what's expected/allowed in this setting.
> >
> > Cheers,
> >
> > Mike
> >
> > PS - I wonder if JSON Schema has the expressiveness for this?
> >
> > On 6/21/17 2:26 AM, Riyafa Abdul Hameed wrote:
> >
> > Hi,
> >
> > I would like to parse the following or any GeoJSON type[1] to a record in
> > AsterixDB:
> > {
> > "type":"Feature",
> > "geometry":{
> > "type":"Point",
> > "coordinates":[
> > -118.40,
> > 33.93
> > ]
> > },
> > "properties":{
> > "code":"LAX",
> > "elevation":38
> > }
> > }
> >
> > The value of properties is optional and is a variable that is it can be
> > any type of object. What is the most suitable datatype to use to
> represent
> > properties in this case?
> > Is something like the following possible?
> >
> > CREATE TYPE GeometryType AS {
> > type: string,
> > geometry: GeometryType,
> > properties: object
> > };
> >
> > I came up with the above because there's a derived type called objects[2]
> > in AsterixDB. The above doesn't work because of the following reasons:
> >
> > - type appears to be keyword
> > - We can't use the defining type within the same type recursively (ie.
> > GeometryType within GeometryType)
> > - The type object cannot be resolved
> >
> > Any suggestions on how a GeoJSON can be parsed into AsterixDB?
> >
> > [1] https://tools.ietf.org/html/rfc7946
> > [2] https://ci.apache.org/projects/asterixdb/datamodel.html#Deri
> > vedTypesObject
> >
> > Thank you
> > Yours sincerely,
> > Riyafa
> >
> >
> >
>
Re: Parse GeoJSON data into a record in AsterixDB
Posted by Ahmed Eldawy <el...@cs.ucr.edu>.
Hi Riyafa,
I think you should use the terms *feature* and *geometry* to avoid
confusion like the following example.
CREATE TYPE AnyObject AS {};
CREATE TYPE *FeatureType* AS {
`type`: string,
the_geom: *geometry*,
properties: AnyObject
};
The internal 'geometry' attribute is what holds the shape geometry while
the outer FeatureType associates additional attributes and properties to
that geometry.
As we discussed in our last call, the first step is to parse GeoJSON as a
regular JSON file and then parse the geometry using a UDF. In this case,
you might have something like:
CREATE TYPE *FeatureType* AS {
`type`: string,
the_geom: *AnyObject*,
properties: AnyObject
};
This will allow you to parse the file without modifying the existing JSON
parser. Then, you can define a UDF called "ParseGeoJSON" which takes as
input the "the_geom" attribute and returns a parsed geometry attribute.
The next step should avoid this additional step and should automatically
detect and parse the geometry attribute directly from GeoJJSON.
Thanks
Ahmed
On Wed, Jun 21, 2017 at 11:40 AM, Yingyi Bu <bu...@gmail.com> wrote:
> >> type appears to be keyword
> `type` would make it valid.
>
> >> We can't use the defining type within the same type recursively (ie.
> GeometryType within GeometryType)
>
> We don't support recursive type definition.
>
> >> The type object cannot be resolved
>
> We don't have a builtin name for a completely open type, but you can
> define one.
>
>
> What you can do is:
>
> CREATE TYPE AnyObject AS {};
>
> CREATE TYPE GeometryType AS {
> `type`: string,
> geometry: SomeType;
> properties: AnyObject
> }
>
>
> Best,
> Yingyi
>
> On Wed, Jun 21, 2017 at 6:33 AM, Mike Carey <dt...@gmail.com> wrote:
>
> > One approach would be to be silent about properties - and then it could
> be
> > there anyway - however, that wouldn't allow you to state the requirement
> > (?) that it must be called properties and/or that it must be an object
> (not
> > a scalar). That could work for now, perhaps? We need to have an "any
> > record type" type name - we've noted a desire for that - unfortunately we
> > don't have one I don't think. I believe the concept is there inside the
> > code, in the type-related areas, but we don't have a keyword like name
> for
> > it. (@Yingyi - comments?) And we do also have a restriction (at the
> type
> > level) that precludes recursion (regular or mutual) in type definitions;
> we
> > probably need to do something about that someday as well.
> >
> > In the meantime, these things could be handled (weakly) by documenting
> > what's expected/allowed in this setting.
> >
> > Cheers,
> >
> > Mike
> >
> > PS - I wonder if JSON Schema has the expressiveness for this?
> >
> > On 6/21/17 2:26 AM, Riyafa Abdul Hameed wrote:
> >
> > Hi,
> >
> > I would like to parse the following or any GeoJSON type[1] to a record in
> > AsterixDB:
> > {
> > "type":"Feature",
> > "geometry":{
> > "type":"Point",
> > "coordinates":[
> > -118.40,
> > 33.93
> > ]
> > },
> > "properties":{
> > "code":"LAX",
> > "elevation":38
> > }
> > }
> >
> > The value of properties is optional and is a variable that is it can be
> > any type of object. What is the most suitable datatype to use to
> represent
> > properties in this case?
> > Is something like the following possible?
> >
> > CREATE TYPE GeometryType AS {
> > type: string,
> > geometry: GeometryType,
> > properties: object
> > };
> >
> > I came up with the above because there's a derived type called objects[2]
> > in AsterixDB. The above doesn't work because of the following reasons:
> >
> > - type appears to be keyword
> > - We can't use the defining type within the same type recursively (ie.
> > GeometryType within GeometryType)
> > - The type object cannot be resolved
> >
> > Any suggestions on how a GeoJSON can be parsed into AsterixDB?
> >
> > [1] https://tools.ietf.org/html/rfc7946
> > [2] https://ci.apache.org/projects/asterixdb/datamodel.html#Deri
> > vedTypesObject
> >
> > Thank you
> > Yours sincerely,
> > Riyafa
> >
> >
> >
>
Re: Parse GeoJSON data into a record in AsterixDB
Posted by Yingyi Bu <bu...@gmail.com>.
>> type appears to be keyword
`type` would make it valid.
>> We can't use the defining type within the same type recursively (ie.
GeometryType within GeometryType)
We don't support recursive type definition.
>> The type object cannot be resolved
We don't have a builtin name for a completely open type, but you can
define one.
What you can do is:
CREATE TYPE AnyObject AS {};
CREATE TYPE GeometryType AS {
`type`: string,
geometry: SomeType;
properties: AnyObject
}
Best,
Yingyi
On Wed, Jun 21, 2017 at 6:33 AM, Mike Carey <dt...@gmail.com> wrote:
> One approach would be to be silent about properties - and then it could be
> there anyway - however, that wouldn't allow you to state the requirement
> (?) that it must be called properties and/or that it must be an object (not
> a scalar). That could work for now, perhaps? We need to have an "any
> record type" type name - we've noted a desire for that - unfortunately we
> don't have one I don't think. I believe the concept is there inside the
> code, in the type-related areas, but we don't have a keyword like name for
> it. (@Yingyi - comments?) And we do also have a restriction (at the type
> level) that precludes recursion (regular or mutual) in type definitions; we
> probably need to do something about that someday as well.
>
> In the meantime, these things could be handled (weakly) by documenting
> what's expected/allowed in this setting.
>
> Cheers,
>
> Mike
>
> PS - I wonder if JSON Schema has the expressiveness for this?
>
> On 6/21/17 2:26 AM, Riyafa Abdul Hameed wrote:
>
> Hi,
>
> I would like to parse the following or any GeoJSON type[1] to a record in
> AsterixDB:
> {
> "type":"Feature",
> "geometry":{
> "type":"Point",
> "coordinates":[
> -118.40,
> 33.93
> ]
> },
> "properties":{
> "code":"LAX",
> "elevation":38
> }
> }
>
> The value of properties is optional and is a variable that is it can be
> any type of object. What is the most suitable datatype to use to represent
> properties in this case?
> Is something like the following possible?
>
> CREATE TYPE GeometryType AS {
> type: string,
> geometry: GeometryType,
> properties: object
> };
>
> I came up with the above because there's a derived type called objects[2]
> in AsterixDB. The above doesn't work because of the following reasons:
>
> - type appears to be keyword
> - We can't use the defining type within the same type recursively (ie.
> GeometryType within GeometryType)
> - The type object cannot be resolved
>
> Any suggestions on how a GeoJSON can be parsed into AsterixDB?
>
> [1] https://tools.ietf.org/html/rfc7946
> [2] https://ci.apache.org/projects/asterixdb/datamodel.html#Deri
> vedTypesObject
>
> Thank you
> Yours sincerely,
> Riyafa
>
>
>
Re: Parse GeoJSON data into a record in AsterixDB
Posted by Yingyi Bu <bu...@gmail.com>.
>> type appears to be keyword
`type` would make it valid.
>> We can't use the defining type within the same type recursively (ie.
GeometryType within GeometryType)
We don't support recursive type definition.
>> The type object cannot be resolved
We don't have a builtin name for a completely open type, but you can
define one.
What you can do is:
CREATE TYPE AnyObject AS {};
CREATE TYPE GeometryType AS {
`type`: string,
geometry: SomeType;
properties: AnyObject
}
Best,
Yingyi
On Wed, Jun 21, 2017 at 6:33 AM, Mike Carey <dt...@gmail.com> wrote:
> One approach would be to be silent about properties - and then it could be
> there anyway - however, that wouldn't allow you to state the requirement
> (?) that it must be called properties and/or that it must be an object (not
> a scalar). That could work for now, perhaps? We need to have an "any
> record type" type name - we've noted a desire for that - unfortunately we
> don't have one I don't think. I believe the concept is there inside the
> code, in the type-related areas, but we don't have a keyword like name for
> it. (@Yingyi - comments?) And we do also have a restriction (at the type
> level) that precludes recursion (regular or mutual) in type definitions; we
> probably need to do something about that someday as well.
>
> In the meantime, these things could be handled (weakly) by documenting
> what's expected/allowed in this setting.
>
> Cheers,
>
> Mike
>
> PS - I wonder if JSON Schema has the expressiveness for this?
>
> On 6/21/17 2:26 AM, Riyafa Abdul Hameed wrote:
>
> Hi,
>
> I would like to parse the following or any GeoJSON type[1] to a record in
> AsterixDB:
> {
> "type":"Feature",
> "geometry":{
> "type":"Point",
> "coordinates":[
> -118.40,
> 33.93
> ]
> },
> "properties":{
> "code":"LAX",
> "elevation":38
> }
> }
>
> The value of properties is optional and is a variable that is it can be
> any type of object. What is the most suitable datatype to use to represent
> properties in this case?
> Is something like the following possible?
>
> CREATE TYPE GeometryType AS {
> type: string,
> geometry: GeometryType,
> properties: object
> };
>
> I came up with the above because there's a derived type called objects[2]
> in AsterixDB. The above doesn't work because of the following reasons:
>
> - type appears to be keyword
> - We can't use the defining type within the same type recursively (ie.
> GeometryType within GeometryType)
> - The type object cannot be resolved
>
> Any suggestions on how a GeoJSON can be parsed into AsterixDB?
>
> [1] https://tools.ietf.org/html/rfc7946
> [2] https://ci.apache.org/projects/asterixdb/datamodel.html#Deri
> vedTypesObject
>
> Thank you
> Yours sincerely,
> Riyafa
>
>
>
Re: Parse GeoJSON data into a record in AsterixDB
Posted by Till Westmann <ti...@apache.org>.
Glancing at RFC 7946 [1] it seems that arbitrary nesting is not allowed.
If I glanced correctly, there are
1) geometry objects and
2) feature objects that can contain a geometry and
3) feature collections that can contain multiple features.
So I believe that recursion should not be a problem.
However, I think that there are other constructs in the structure of a
geometry object that are not expressible in ADM types today - so I
think that we’ll need to work with (at least partially) open types.
Cheers,
Till
[1] https://tools.ietf.org/html/rfc7946
On 21 Jun 2017, at 15:33, Mike Carey wrote:
> One approach would be to be silent about properties - and then it
> could be there anyway - however, that wouldn't allow you to state the
> requirement (?) that it must be called properties and/or that it must
> be an object (not a scalar). That could work for now, perhaps? We
> need to have an "any record type" type name - we've noted a desire for
> that - unfortunately we don't have one I don't think. I believe the
> concept is there inside the code, in the type-related areas, but we
> don't have a keyword like name for it. (@Yingyi - comments?) And we
> do also have a restriction (at the type level) that precludes
> recursion (regular or mutual) in type definitions; we probably need to
> do something about that someday as well.
>
> In the meantime, these things could be handled (weakly) by documenting
> what's expected/allowed in this setting.
>
> Cheers,
>
> Mike
>
> PS - I wonder if JSON Schema has the expressiveness for this?
>
>
> On 6/21/17 2:26 AM, Riyafa Abdul Hameed wrote:
>> Hi,
>>
>> I would like to parse the following or any GeoJSON type[1] to a
>> record in AsterixDB:
>> {
>> "type":"Feature",
>> "geometry":{
>> "type":"Point",
>> "coordinates":[
>> -118.40,
>> 33.93
>> ]
>> },
>> "properties":{
>> "code":"LAX",
>> "elevation":38
>> }
>> }
>>
>> The value of properties is optional and is a variable that is it can
>> be any type of object. What is the most suitable datatype to use to
>> represent properties in this case?
>> Is something like the following possible?
>>
>> CREATE TYPE GeometryType AS {
>> type: string,
>> geometry: GeometryType,
>> properties: object
>> };
>>
>> I came up with the above because there's a derived type called
>> objects[2] in AsterixDB. The above doesn't work because of the
>> following reasons:
>>
>> * type appears to be keyword
>> * We can't use the defining type within the same type recursively
>> (ie. GeometryType within GeometryType)
>> * The type object cannot be resolved
>>
>> Any suggestions on how a GeoJSON can be parsed into AsterixDB?
>>
>> [1] https://tools.ietf.org/html/rfc7946
>> [2]
>> https://ci.apache.org/projects/asterixdb/datamodel.html#DerivedTypesObject
>>
>> Thank you
>> Yours sincerely,
>> Riyafa
>>
Re: Parse GeoJSON data into a record in AsterixDB
Posted by Mike Carey <dt...@gmail.com>.
One approach would be to be silent about properties - and then it could
be there anyway - however, that wouldn't allow you to state the
requirement (?) that it must be called properties and/or that it must be
an object (not a scalar). That could work for now, perhaps? We need to
have an "any record type" type name - we've noted a desire for that -
unfortunately we don't have one I don't think. I believe the concept is
there inside the code, in the type-related areas, but we don't have a
keyword like name for it. (@Yingyi - comments?) And we do also have a
restriction (at the type level) that precludes recursion (regular or
mutual) in type definitions; we probably need to do something about that
someday as well.
In the meantime, these things could be handled (weakly) by documenting
what's expected/allowed in this setting.
Cheers,
Mike
PS - I wonder if JSON Schema has the expressiveness for this?
On 6/21/17 2:26 AM, Riyafa Abdul Hameed wrote:
> Hi,
>
> I would like to parse the following or any GeoJSON type[1] to a record
> in AsterixDB:
> {
> "type":"Feature",
> "geometry":{
> "type":"Point",
> "coordinates":[
> -118.40,
> 33.93
> ]
> },
> "properties":{
> "code":"LAX",
> "elevation":38
> }
> }
>
> The value of properties is optional and is a variable that is it can
> be any type of object. What is the most suitable datatype to use to
> represent properties in this case?
> Is something like the following possible?
>
> CREATE TYPE GeometryType AS {
> type: string,
> geometry: GeometryType,
> properties: object
> };
>
> I came up with the above because there's a derived type called
> objects[2] in AsterixDB. The above doesn't work because of the
> following reasons:
>
> * type appears to be keyword
> * We can't use the defining type within the same type recursively
> (ie. GeometryType within GeometryType)
> * The type object cannot be resolved
>
> Any suggestions on how a GeoJSON can be parsed into AsterixDB?
>
> [1] https://tools.ietf.org/html/rfc7946
> [2]
> https://ci.apache.org/projects/asterixdb/datamodel.html#DerivedTypesObject
>
> Thank you
> Yours sincerely,
> Riyafa
>
Re: Parse GeoJSON data into a record in AsterixDB
Posted by Mike Carey <dt...@gmail.com>.
One approach would be to be silent about properties - and then it could
be there anyway - however, that wouldn't allow you to state the
requirement (?) that it must be called properties and/or that it must be
an object (not a scalar). That could work for now, perhaps? We need to
have an "any record type" type name - we've noted a desire for that -
unfortunately we don't have one I don't think. I believe the concept is
there inside the code, in the type-related areas, but we don't have a
keyword like name for it. (@Yingyi - comments?) And we do also have a
restriction (at the type level) that precludes recursion (regular or
mutual) in type definitions; we probably need to do something about that
someday as well.
In the meantime, these things could be handled (weakly) by documenting
what's expected/allowed in this setting.
Cheers,
Mike
PS - I wonder if JSON Schema has the expressiveness for this?
On 6/21/17 2:26 AM, Riyafa Abdul Hameed wrote:
> Hi,
>
> I would like to parse the following or any GeoJSON type[1] to a record
> in AsterixDB:
> {
> "type":"Feature",
> "geometry":{
> "type":"Point",
> "coordinates":[
> -118.40,
> 33.93
> ]
> },
> "properties":{
> "code":"LAX",
> "elevation":38
> }
> }
>
> The value of properties is optional and is a variable that is it can
> be any type of object. What is the most suitable datatype to use to
> represent properties in this case?
> Is something like the following possible?
>
> CREATE TYPE GeometryType AS {
> type: string,
> geometry: GeometryType,
> properties: object
> };
>
> I came up with the above because there's a derived type called
> objects[2] in AsterixDB. The above doesn't work because of the
> following reasons:
>
> * type appears to be keyword
> * We can't use the defining type within the same type recursively
> (ie. GeometryType within GeometryType)
> * The type object cannot be resolved
>
> Any suggestions on how a GeoJSON can be parsed into AsterixDB?
>
> [1] https://tools.ietf.org/html/rfc7946
> [2]
> https://ci.apache.org/projects/asterixdb/datamodel.html#DerivedTypesObject
>
> Thank you
> Yours sincerely,
> Riyafa
>