You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sis.apache.org by Adam Estrada <es...@gmail.com> on 2015/09/03 14:47:16 UTC
Re: [gdal-dev] Store also native geometry into Elastic Search?
Even,
I think the reason to store everything by default was just to be on
the safe side and it satisfied some of the project requirements at the
time. That was back in 2011 so ElasticSearch and this driver are
probably not in sync any longer. There is a lot of work that can be
done to the driver to get up to date with all the cool stuff that is
now available in ES. I'd be interested in starting a roadmap
specifically for the ES driver starting with a read capability and the
ability to store other geometry types.
Adam
On Thu, Sep 3, 2015 at 7:42 AM, Even Rouault <ev...@spatialys.com> wrote:
> Le jeudi 03 septembre 2015 12:49:30, Jukka Rahkonen a écrit :
>> Hi,
>>
>> What if somebody would like to put and deliver some official spatial data
>> (cadastral data) which are not in EPSG:4326 with Elastic Search? Could it
>> be possible to translate the geometries into EPSG:4326 and store and index
>> them as geo_shape, but also keep the native geometries and save them into
>> another field? Perhaps json does not like to have many geometry fields but
>> how about saving the native geometry as WKT? Indexing WKT field feels
>> stupid but maybe ES could be mapped to skip it. Reasoning is that
>> conversion from native to EPSG:4326 and back to native is not necessary
>> accurate and coordinates may change a bit.
>
> Jukka (and little question for you Adam if you read this),
>
> Well, you can do that with a Spatialite SQL query like :
>
> -sql "select *, ST_AsText(geometry) as WKT_32631 from simple_EPSG_32631" -dialect sqlite
>
> As far as not indexing the WKT_32631 field, you would need to:
> 1) export the default mapping with -lco WRITE_MAPPING=mapping.json
> 2) add "index":"no" to the WKT_32631 definition (https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-core-types.html)
> 3) run again with -lco MAPPING=mapping.json
>
> Doing all the above steps in a fully automated way, perhaps with a dedicated PRESERVE_SRC_GEOMETRY
> layer creation option, would certainly be possible, but personnaly, it's more in the
> something-to-perhaps-consider-someday queue ;-)
>
> Perhaps we would need a few creation options to alter more easily the mapping
> of the created fields (rather than the above export/modify/import workflow). Not sure
> at this point which ones would make sense, and if we aren't careful, there's a risk we end
> up with something complicated that would be redundant with the JSon API.
> I also see that the existing write-only driver creates a mapping that explicitely stores fields,
> and as we don't disable storing the _source field, so apparently this wouldn't be necessary.
> Hum, might be wise to remove the explicit "store": "yes"
>
> @adam : do you remind the reason for adding "store": "yes" in the generated ES mapping ?
>
>>
>> Am I even right that the conversion into EPSG:4326 is necessary for using
>> the geo_shape query?
>
> That's my understanding of https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo-shape-type.html
> "In GeoJSON, and therefore Elasticsearch, the correct coordinate order is longitude, latitude (X, Y) within coordinate arrays"
>
> Actually I've just tried to manually insert with invalidate lon,lat and the behaviour is
> hard to decypher:
>
> e.g
>
> curl -d '{ "geometry": { "type": "linestring", "coordinates": [ [ 800.0, 0.0 ], [ 3.0, 50.0 ] ] } }' "http://localhost:9200/test/FeatureCollection?pretty"
>
> works,
>
> but
>
> curl -d '{ "geometry": { "type": "linestring", "coordinates": [ [ 1000.0, 0.0 ], [ 3.0, 50.0 ] ] } }' "http://localhost:9200/test/FeatureCollection?pretty"
>
> throws an error:
> "MapperParsingException[failed to parse [geometry]]; nested: IllegalArgumentException[This method does not support GeometryCollection arguments]; "
>
> And I assume that any spherical geospatial search will behave in undefined ways if you insert
> coordinates that are not (lon,lat)
>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
Re: [gdal-dev] Store also native geometry into Elastic Search?
Posted by Even Rouault <ev...@spatialys.com>.
Le jeudi 03 septembre 2015 15:04:13, Adam Estrada a écrit :
> Even,
>
> This would have been around version 0.16.x. Not really sure which one
> exactly but I don't think any changes to the geometry support happened
> until 2012-2013?
Ah sorry, my fault for embedding you in a thread with a dedicated topic in its
title and my question being more general ;-) I was refering to storing regular fields.
I found this thread ( http://stackoverflow.com/questions/17103047/why-do-i-need-storeyes-in-elasticsearch )
where it is said that storing the field might be more performant in some cases
(avoid fetching the whole document and parse it), but in other cases not.
--
Spatialys - Geospatial professional services
http://www.spatialys.com
Re: [gdal-dev] Store also native geometry into Elastic Search?
Posted by Adam Estrada <es...@gmail.com>.
Even,
This would have been around version 0.16.x. Not really sure which one
exactly but I don't think any changes to the geometry support happened
until 2012-2013?
VERY cool about all the modifications you guys have made! This is very exciting!
http://gdal.org/drv_elasticsearch.html
Adam
On Thu, Sep 3, 2015 at 8:52 AM, Even Rouault <ev...@spatialys.com> wrote:
> Le jeudi 03 septembre 2015 14:47:16, Adam Estrada a écrit :
>> Even,
>>
>> I think the reason to store everything by default was just to be on
>> the safe side and it satisfied some of the project requirements at the
>> time. That was back in 2011 so ElasticSearch and this driver are
>> probably not in sync any longer.
>
> OK, do you remember which ES version was targetted at that time ? So I can
> potentially be able to find in the doc the behaviour and see if it has changed.
>
>> There is a lot of work that can be
>> done to the driver to get up to date with all the cool stuff that is
>> now available in ES. I'd be interested in starting a roadmap
>> specifically for the ES driver starting with a read capability and the
>> ability to store other geometry types.
>
> Actually, you should have a look at the recent additions at
> http://gdal.org/drv_elasticsearch.html ;-)
>
>>
>> Adam
>>
>> On Thu, Sep 3, 2015 at 7:42 AM, Even Rouault <ev...@spatialys.com>
> wrote:
>> > Le jeudi 03 septembre 2015 12:49:30, Jukka Rahkonen a écrit :
>> >> Hi,
>> >>
>> >> What if somebody would like to put and deliver some official spatial
>> >> data (cadastral data) which are not in EPSG:4326 with Elastic Search?
>> >> Could it be possible to translate the geometries into EPSG:4326 and
>> >> store and index them as geo_shape, but also keep the native geometries
>> >> and save them into another field? Perhaps json does not like to have
>> >> many geometry fields but how about saving the native geometry as WKT?
>> >> Indexing WKT field feels stupid but maybe ES could be mapped to skip
>> >> it. Reasoning is that conversion from native to EPSG:4326 and back to
>> >> native is not necessary accurate and coordinates may change a bit.
>> >
>> > Jukka (and little question for you Adam if you read this),
>> >
>> > Well, you can do that with a Spatialite SQL query like :
>> >
>> > -sql "select *, ST_AsText(geometry) as WKT_32631 from simple_EPSG_32631"
>> > -dialect sqlite
>> >
>> > As far as not indexing the WKT_32631 field, you would need to:
>> > 1) export the default mapping with -lco WRITE_MAPPING=mapping.json
>> > 2) add "index":"no" to the WKT_32631 definition
>> > (https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping
>> > -core-types.html) 3) run again with -lco MAPPING=mapping.json
>> >
>> > Doing all the above steps in a fully automated way, perhaps with a
>> > dedicated PRESERVE_SRC_GEOMETRY layer creation option, would certainly
>> > be possible, but personnaly, it's more in the
>> > something-to-perhaps-consider-someday queue ;-)
>> >
>> > Perhaps we would need a few creation options to alter more easily the
>> > mapping of the created fields (rather than the above
>> > export/modify/import workflow). Not sure at this point which ones would
>> > make sense, and if we aren't careful, there's a risk we end up with
>> > something complicated that would be redundant with the JSon API. I also
>> > see that the existing write-only driver creates a mapping that
>> > explicitely stores fields, and as we don't disable storing the _source
>> > field, so apparently this wouldn't be necessary. Hum, might be wise to
>> > remove the explicit "store": "yes"
>> >
>> > @adam : do you remind the reason for adding "store": "yes" in the
>> > generated ES mapping ?
>> >
>> >> Am I even right that the conversion into EPSG:4326 is necessary for
>> >> using the geo_shape query?
>> >
>> > That's my understanding of
>> > https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-
>> > geo-shape-type.html "In GeoJSON, and therefore Elasticsearch, the correct
>> > coordinate order is longitude, latitude (X, Y) within coordinate arrays"
>> >
>> > Actually I've just tried to manually insert with invalidate lon,lat and
>> > the behaviour is hard to decypher:
>> >
>> > e.g
>> >
>> > curl -d '{ "geometry": { "type": "linestring", "coordinates": [ [ 800.0,
>> > 0.0 ], [ 3.0, 50.0 ] ] } }'
>> > "http://localhost:9200/test/FeatureCollection?pretty"
>> >
>> > works,
>> >
>> > but
>> >
>> > curl -d '{ "geometry": { "type": "linestring", "coordinates": [ [
>> > 1000.0, 0.0 ], [ 3.0, 50.0 ] ] } }'
>> > "http://localhost:9200/test/FeatureCollection?pretty"
>> >
>> > throws an error:
>> > "MapperParsingException[failed to parse [geometry]]; nested:
>> > IllegalArgumentException[This method does not support GeometryCollection
>> > arguments]; "
>> >
>> > And I assume that any spherical geospatial search will behave in
>> > undefined ways if you insert coordinates that are not (lon,lat)
>> >
>> > Even
>> >
>> > --
>> > Spatialys - Geospatial professional services
>> > http://www.spatialys.com
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com