You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Christopher Schmidt <fa...@googlemail.com> on 2010/11/03 20:16:34 UTC
Rethinking spatial implementation
Hi all,
I saw a mail thread "Rethinking Cartesian Tiers implementation"
(here<http://www.mail-archive.com/dev@lucene.apache.org/msg03593.html>
).
Is there any work in progress regarding this? If yes, is the current
implementation deprecated or do you plan some enhancements (other
projections or spatial indexes) ?
I am asking because I want to use Lucene's spatial indexing in a production
system...
--
Christopher
twitter: @fakod
blog: http://blog.fakod.eu
Re: Rethinking spatial implementation
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi All,
FYI, Apache SIS [1], currently Incubating, is working on building an ASLv2 licensed library comparable to JTS or GeoTools. You'll notice that most of the GIS related libs out there are GPL or LGPL (or at least I did), so I decided to do something about it.
If anyone else is interested in joining the cause, we'd welcome you over there.
At present, we have code that implements a QuadTree storage and does PointRadius and bounding box computations, as well as a REST-ful web service to handle spatial location based on those 2 methods. We're close to making an 0.1-incubating release.
Cheers,
Chris
[1] http://incubator.apache.org/sis/
On 11/8/10 2:40 AM, "Chris Male" <ge...@gmail.com> wrote:
Hi,
I'll jump in and give my opinion:
Can you clarify what you mean with "the Sinusoidal projection is broken"?
Inside Spatial Lucene's Cartesian codebase is an implementation of Sinusoidal projection. Grant discovered while working on improving the testing coverage of the code that the implementation doesn't actually match the formula specified on Wikipedia. When we tried to change it, many tests broke since the overall logic somehow depends on this broken implementation.
Would it be possible to use a LGPL library like the Java Topology Suite (JTS: http://www.vividsolutions.com/jts/JTSHome.htm)?
This is something we've talked about using. I think it would be nice to offload some of the geographic-specific from Lucene. So using another library would be good. At the same time it limits our options for optimizations and the like. I'm certainly looking into it though.
Thanks,
Chris
Neo4j is using JTS for creating a spatial index (code is here: https://github.com/neo4j/neo4j-spatial)...
(I've just seen that JTS has some index creation classes, but I'm not at all familiar with them)
Christopher
On Mon, Nov 8, 2010 at 1:10 AM, Grant Ingersoll <gs...@apache.org> wrote:
On Nov 6, 2010, at 5:23 PM, Christopher Schmidt wrote:
> Hi Ryan, thx for your answer.
>
> You mean there is room for improvement and volunteers?
We've been looking at replacing it with the Military Grid system. The primary issue with the current is that the Sinusoidal projection is broken which then breaks almost all the tests. I worked on it for a while trying to straighten it out, but gave up and now think it is easier to implement clean. I definitely would like to see a tier/grid implementation.
>
> On Friday, November 5, 2010, Ryan McKinley <ry...@gmail.com> wrote:
>> Hi Christopher -
>>
>> I do not believe there is any active work on this. From what I
>> understand, the Tier implementation works OK within some constraints,
>> but we could not get it to pass more robust testing that the other
>> methods were using.
>>
>> However, LatLonType and GeoHashField are well tested and work well --
>> the Tier type may have better performance when your index is really
>> large, but no active developers understand it and no-one has stepped
>> up to figure it out.
>>
>> ryan
>>
>>
>> On Wed, Nov 3, 2010 at 3:16 PM, Christopher Schmidt
>> <fa...@googlemail.com> wrote:
>>> Hi all,
>>> I saw a mail thread "Rethinking Cartesian Tiers implementation" (here).
>>> Is there any work in progress regarding this? If yes, is the current
>>> implementation deprecated or do you plan some enhancements (other
>>> projections or spatial indexes) ?
>>> I am asking because I want to use Lucene's spatial indexing in a production
>>> system...
>>>
>>> --
>>> Christopher
>>> twitter: @fakod
>>> blog: http://blog.fakod.eu
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>
> --
> Christopher
> twitter: @fakod
> blog: http://blog.fakod.eu
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem docs using Solr/Lucene:
http://www.lucidimagination.com/search
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Rethinking spatial implementation
Posted by Chris Male <ge...@gmail.com>.
Hi,
I'll jump in and give my opinion:
Can you clarify what you mean with "the Sinusoidal projection is broken"?
>
Inside Spatial Lucene's Cartesian codebase is an implementation of
Sinusoidal projection. Grant discovered while working on improving the
testing coverage of the code that the implementation doesn't actually match
the formula specified on Wikipedia. When we tried to change it, many tests
broke since the overall logic somehow depends on this broken implementation.
>
> Would it be possible to use a LGPL library like the Java Topology Suite
> (JTS: http://www.vividsolutions.com/jts/JTSHome.htm)?
>
This is something we've talked about using. I think it would be nice to
offload some of the geographic-specific from Lucene. So using another
library would be good. At the same time it limits our options for
optimizations and the like. I'm certainly looking into it though.
Thanks,
Chris
>
> Neo4j is using JTS for creating a spatial index (code is here:
> https://github.com/neo4j/neo4j-spatial)...
>
> (I've just seen that JTS has some index creation classes, but I'm not at
> all familiar with them)
>
> Christopher
>
>
> On Mon, Nov 8, 2010 at 1:10 AM, Grant Ingersoll <gs...@apache.org>wrote:
>
>>
>> On Nov 6, 2010, at 5:23 PM, Christopher Schmidt wrote:
>>
>> > Hi Ryan, thx for your answer.
>> >
>> > You mean there is room for improvement and volunteers?
>>
>> We've been looking at replacing it with the Military Grid system. The
>> primary issue with the current is that the Sinusoidal projection is broken
>> which then breaks almost all the tests. I worked on it for a while trying
>> to straighten it out, but gave up and now think it is easier to implement
>> clean. I definitely would like to see a tier/grid implementation.
>>
>>
>> >
>> > On Friday, November 5, 2010, Ryan McKinley <ry...@gmail.com> wrote:
>> >> Hi Christopher -
>> >>
>> >> I do not believe there is any active work on this. From what I
>> >> understand, the Tier implementation works OK within some constraints,
>> >> but we could not get it to pass more robust testing that the other
>> >> methods were using.
>> >>
>> >> However, LatLonType and GeoHashField are well tested and work well --
>> >> the Tier type may have better performance when your index is really
>> >> large, but no active developers understand it and no-one has stepped
>> >> up to figure it out.
>> >>
>> >> ryan
>> >>
>> >>
>> >> On Wed, Nov 3, 2010 at 3:16 PM, Christopher Schmidt
>> >> <fa...@googlemail.com> wrote:
>> >>> Hi all,
>> >>> I saw a mail thread "Rethinking Cartesian Tiers implementation"
>> (here).
>> >>> Is there any work in progress regarding this? If yes, is the current
>> >>> implementation deprecated or do you plan some enhancements (other
>> >>> projections or spatial indexes) ?
>> >>> I am asking because I want to use Lucene's spatial indexing in a
>> production
>> >>> system...
>> >>>
>> >>> --
>> >>> Christopher
>> >>> twitter: @fakod
>> >>> blog: http://blog.fakod.eu
>> >>>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: dev-help@lucene.apache.org
>> >>
>> >>
>> >
>> > --
>> > Christopher
>> > twitter: @fakod
>> > blog: http://blog.fakod.eu
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: dev-help@lucene.apache.org
>> >
>>
>> --------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com/
>>
>> Search the Lucene ecosystem docs using Solr/Lucene:
>> http://www.lucidimagination.com/search
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>
>
> --
> Christopher
> twitter: @fakod
> blog: http://blog.fakod.eu
>
>
--
Chris Male | Software Developer | JTeam BV.| www.jteam.nl
Re: Rethinking spatial implementation
Posted by Ryan McKinley <ry...@gmail.com>.
> Neo4j is using JTS for creating a spatial index (code is here:
> https://github.com/neo4j/neo4j-spatial)...
> (I've just seen that JTS has some index creation classes, but I'm not at all
> familiar with them)
JTS does not have a spatial index -- it is good for spatial operations
(check if some shape is within/intersects/etc another shape) In
Neo4j, they use JTS to build an RTree that is stored in their native
graph format:
https://github.com/neo4j/neo4j-spatial/blob/master/src/main/java/org/neo4j/gis/spatial/RTreeIndex.java
Building an RTree in lucene is a bit more difficult since we can not
easily update the value of a given field. I'd like to figure some way
to do this though.
ryan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Rethinking spatial implementation
Posted by Christopher Schmidt <fa...@googlemail.com>.
Some questions:
@Grant: Can you clarify what you mean with "the Sinusoidal projection is
broken"?
Would it be possible to use a LGPL library like the Java Topology Suite
(JTS: http://www.vividsolutions.com/jts/JTSHome.htm)?
Neo4j is using JTS for creating a spatial index (code is here:
https://github.com/neo4j/neo4j-spatial)...
(I've just seen that JTS has some index creation classes, but I'm not at all
familiar with them)
Christopher
On Mon, Nov 8, 2010 at 1:10 AM, Grant Ingersoll <gs...@apache.org> wrote:
>
> On Nov 6, 2010, at 5:23 PM, Christopher Schmidt wrote:
>
> > Hi Ryan, thx for your answer.
> >
> > You mean there is room for improvement and volunteers?
>
> We've been looking at replacing it with the Military Grid system. The
> primary issue with the current is that the Sinusoidal projection is broken
> which then breaks almost all the tests. I worked on it for a while trying
> to straighten it out, but gave up and now think it is easier to implement
> clean. I definitely would like to see a tier/grid implementation.
>
>
> >
> > On Friday, November 5, 2010, Ryan McKinley <ry...@gmail.com> wrote:
> >> Hi Christopher -
> >>
> >> I do not believe there is any active work on this. From what I
> >> understand, the Tier implementation works OK within some constraints,
> >> but we could not get it to pass more robust testing that the other
> >> methods were using.
> >>
> >> However, LatLonType and GeoHashField are well tested and work well --
> >> the Tier type may have better performance when your index is really
> >> large, but no active developers understand it and no-one has stepped
> >> up to figure it out.
> >>
> >> ryan
> >>
> >>
> >> On Wed, Nov 3, 2010 at 3:16 PM, Christopher Schmidt
> >> <fa...@googlemail.com> wrote:
> >>> Hi all,
> >>> I saw a mail thread "Rethinking Cartesian Tiers implementation" (here).
> >>> Is there any work in progress regarding this? If yes, is the current
> >>> implementation deprecated or do you plan some enhancements (other
> >>> projections or spatial indexes) ?
> >>> I am asking because I want to use Lucene's spatial indexing in a
> production
> >>> system...
> >>>
> >>> --
> >>> Christopher
> >>> twitter: @fakod
> >>> blog: http://blog.fakod.eu
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: dev-help@lucene.apache.org
> >>
> >>
> >
> > --
> > Christopher
> > twitter: @fakod
> > blog: http://blog.fakod.eu
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
> >
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
>
> Search the Lucene ecosystem docs using Solr/Lucene:
> http://www.lucidimagination.com/search
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
--
Christopher
twitter: @fakod
blog: http://blog.fakod.eu
Re: Rethinking spatial implementation
Posted by Grant Ingersoll <gs...@apache.org>.
On Nov 6, 2010, at 5:23 PM, Christopher Schmidt wrote:
> Hi Ryan, thx for your answer.
>
> You mean there is room for improvement and volunteers?
We've been looking at replacing it with the Military Grid system. The primary issue with the current is that the Sinusoidal projection is broken which then breaks almost all the tests. I worked on it for a while trying to straighten it out, but gave up and now think it is easier to implement clean. I definitely would like to see a tier/grid implementation.
>
> On Friday, November 5, 2010, Ryan McKinley <ry...@gmail.com> wrote:
>> Hi Christopher -
>>
>> I do not believe there is any active work on this. From what I
>> understand, the Tier implementation works OK within some constraints,
>> but we could not get it to pass more robust testing that the other
>> methods were using.
>>
>> However, LatLonType and GeoHashField are well tested and work well --
>> the Tier type may have better performance when your index is really
>> large, but no active developers understand it and no-one has stepped
>> up to figure it out.
>>
>> ryan
>>
>>
>> On Wed, Nov 3, 2010 at 3:16 PM, Christopher Schmidt
>> <fa...@googlemail.com> wrote:
>>> Hi all,
>>> I saw a mail thread "Rethinking Cartesian Tiers implementation" (here).
>>> Is there any work in progress regarding this? If yes, is the current
>>> implementation deprecated or do you plan some enhancements (other
>>> projections or spatial indexes) ?
>>> I am asking because I want to use Lucene's spatial indexing in a production
>>> system...
>>>
>>> --
>>> Christopher
>>> twitter: @fakod
>>> blog: http://blog.fakod.eu
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>
> --
> Christopher
> twitter: @fakod
> blog: http://blog.fakod.eu
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem docs using Solr/Lucene:
http://www.lucidimagination.com/search
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Rethinking spatial implementation
Posted by Ryan McKinley <ry...@gmail.com>.
On Sat, Nov 6, 2010 at 5:23 PM, Christopher Schmidt
<fa...@googlemail.com> wrote:
> Hi Ryan, thx for your answer.
>
> You mean there is room for improvement and volunteers?
For sure!
Patches and contributions are always welcome -- I am starting to have
more time, so will hopefully be able to help as well.
Ryan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Rethinking spatial implementation
Posted by Christopher Schmidt <fa...@googlemail.com>.
Hi Ryan, thx for your answer.
You mean there is room for improvement and volunteers?
On Friday, November 5, 2010, Ryan McKinley <ry...@gmail.com> wrote:
> Hi Christopher -
>
> I do not believe there is any active work on this. From what I
> understand, the Tier implementation works OK within some constraints,
> but we could not get it to pass more robust testing that the other
> methods were using.
>
> However, LatLonType and GeoHashField are well tested and work well --
> the Tier type may have better performance when your index is really
> large, but no active developers understand it and no-one has stepped
> up to figure it out.
>
> ryan
>
>
> On Wed, Nov 3, 2010 at 3:16 PM, Christopher Schmidt
> <fa...@googlemail.com> wrote:
>> Hi all,
>> I saw a mail thread "Rethinking Cartesian Tiers implementation" (here).
>> Is there any work in progress regarding this? If yes, is the current
>> implementation deprecated or do you plan some enhancements (other
>> projections or spatial indexes) ?
>> I am asking because I want to use Lucene's spatial indexing in a production
>> system...
>>
>> --
>> Christopher
>> twitter: @fakod
>> blog: http://blog.fakod.eu
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
--
Christopher
twitter: @fakod
blog: http://blog.fakod.eu
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Rethinking spatial implementation
Posted by Ryan McKinley <ry...@gmail.com>.
Hi Christopher -
I do not believe there is any active work on this. From what I
understand, the Tier implementation works OK within some constraints,
but we could not get it to pass more robust testing that the other
methods were using.
However, LatLonType and GeoHashField are well tested and work well --
the Tier type may have better performance when your index is really
large, but no active developers understand it and no-one has stepped
up to figure it out.
ryan
On Wed, Nov 3, 2010 at 3:16 PM, Christopher Schmidt
<fa...@googlemail.com> wrote:
> Hi all,
> I saw a mail thread "Rethinking Cartesian Tiers implementation" (here).
> Is there any work in progress regarding this? If yes, is the current
> implementation deprecated or do you plan some enhancements (other
> projections or spatial indexes) ?
> I am asking because I want to use Lucene's spatial indexing in a production
> system...
>
> --
> Christopher
> twitter: @fakod
> blog: http://blog.fakod.eu
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org