You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Peter Lancaster <pe...@findmypast.com> on 2018/12/13 11:38:07 UTC

Geofilt and distance measurement problems using SpatialRecursivePrefixTreeFieldType field type

I am currently using Solr 5.5.2 and implementing a GeoSpatial search that returns results within a radius in Km of a specified LatLon. Using a field of type solr.LatLonType and a geofilt query this gives good results but is much slower than our regular queries. Using a bbox query is faster but of course less accurate.

I then attempted to use a field of type solr.SpatialRecursivePrefixTreeFieldType to check performance and because I want to be able to do searches within a polygon eventually. The field is defined as follows

<fieldType name="location_rpt"   class="solr.SpatialRecursivePrefixTreeFieldType" spatialContextFactory="com.spatial4j.core.context.jts.JtsSpatialContextFactory"
            geo="true" distErrPct="0.05" maxDistErr="0.2" distanceUnits="kilometers" autoIndex="true"/>

<dynamicField name="*__location_rpt" type="location_rpt" indexed="true" stored="true" multiValued="false" omitNorms="true" />

I'm just using it to index single points right now. The problem is that the distance calculation is not working correctly. It seems to overstate the distances for differences in longitude.

For example a query for &fl=Id,LatLonRPT__location_rpt,_dist_:geodist()&sfield=LatLonRPT__location_rpt&pt=53.409490 -2.979677&query={!geofilt sfield=LatLonRPT__location_rpt pt="53.409490 -2.979677" d=25} returns

{
                "Id": "HAR/CH1/80763270",
                "LatLonRPT__location_rpt": "53.2 -2.916666",
                "_dist_": 24.295607
},
{
                "Id": "HAR/CH42/1918283949",
                "LatLonRPT__location_rpt": "53.393239 -3.028859",
                "_dist_": 5.7587695
}

The true distances for these results are 23.67 and 3.73 km and other results at a true distance of 17 km aren't returned within the 25 km radius.

The explain has the following
                +IntersectsPrefixTreeQuery(IntersectsPrefixTreeQuery(fieldName=LatLonRPT__location_rpt,queryShape=Circle(Pt(x=53.40949,y=-2.979677), d=0.2° 25.00km),detailLevel=6,prefixGridScanLevel=7))

Is my set up incorrect in some way or is the SpatialRecursivePrefixTreeFieldType not suitable for doing radius searches on points in this way?

Thanks in anticipation for any suggestions.

Peter Lancaster.

________________________________
This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of findmypast shall be understood as neither given nor endorsed by it.
________________________________

__________________________________________________________________________

This email has been checked for virus and other malicious content prior to leaving our network.
__________________________________________________________________________

RE: Geofilt and distance measurement problems using SpatialRecursivePrefixTreeFieldType field type

Posted by Peter Lancaster <pe...@findmypast.com>.
Hi David,

Thanks for coming back to me.

When using rpt fields I believe you do need to use a space between Lat and Lon to indicate a point; for rpt fields commas are used to separate points in a polygon. See https://archive.apache.org/dist/lucene/solr/ref-guide/apache-solr-ref-guide-5.5.pdf bottom of page 364.

We do have the Lat Lons the correct way round and the calculations work just fine when using a LatLonType it's just very slow which is why I was trying out the rpt field type to see how this compared. Yes it's faster using rpt but the distance calculations just don't seem to work.

Of course it's not just the distance calculation that's not working the filtering doesn't work correctly as a consequence so e.g. fq={!geofilt sfield=LatLonRPT__location_rpt pt="53.409490 -2.979677" d=25}  omits Lat Lons that are say only 17Km away. It appears that it's just not using a correct haversine calculation for distances.

For the moment I'm having to use a point type to filter on because rpt doesn't calculate distances correctly, and I'm having to use a bounding box query rather than a radius because the performance isn't good enough with radius.

Just wondering if there's something else wrong with our set up which is causing this behaviour for rpt fields.

Cheers,
Peter.


-----Original Message-----
From: David Smiley [mailto:david.w.smiley@gmail.com]
Sent: 21 December 2018 04:44
To: solr-user@lucene.apache.org
Subject: Re: Geofilt and distance measurement problems using SpatialRecursivePrefixTreeFieldType field type

Hi Peter,

Use of an RPT field for distance sorting/boosting is to be avoided where possible because it's very inefficient at this specific use-case.  Simply use LatLonType for this task, and continue to use RPT for the filter/search use-case.

Also I see you putting a space between the coordinates instead of a
comma...   yet you have geo (latitude & longitude data) so this is a bit
confusing.  Do "lat,lon".  I think a space will be interpreted as "x y"
(thus reversed).  Perhaps you've mixed up the coordinates and this explains the error?  A quick lookup of your sample coordinates suggests to me this is likely the problem.  It's a common mistake.

BTW this:
maxDistErr="0.2" distanceUnits="kilometers"
means 200m accuracy (or better).  Is this what you want?  Just checking.

~ David

On Thu, Dec 13, 2018 at 6:38 AM Peter Lancaster < peter.lancaster@findmypast.com> wrote:

> I am currently using Solr 5.5.2 and implementing a GeoSpatial search
> that returns results within a radius in Km of a specified LatLon.
> Using a field of type solr.LatLonType and a geofilt query this gives
> good results but is much slower than our regular queries. Using a bbox
> query is faster but of course less accurate.
>
> I then attempted to use a field of type
> solr.SpatialRecursivePrefixTreeFieldType to check performance and
> because I want to be able to do searches within a polygon eventually.
> The field is defined as follows
>
> <fieldType name="location_rpt"
>  class="solr.SpatialRecursivePrefixTreeFieldType"
> spatialContextFactory="com.spatial4j.core.context.jts.JtsSpatialContextFactory"
>             geo="true" distErrPct="0.05" maxDistErr="0.2"
> distanceUnits="kilometers" autoIndex="true"/>
>
> <dynamicField name="*__location_rpt" type="location_rpt" indexed="true"
> stored="true" multiValued="false" omitNorms="true" />
>
> I'm just using it to index single points right now. The problem is
> that the distance calculation is not working correctly. It seems to
> overstate the distances for differences in longitude.
>
> For example a query for
> &fl=Id,LatLonRPT__location_rpt,_dist_:geodist()&sfield=LatLonRPT__loca
> tion_rpt&pt=53.409490 -2.979677&query={!geofilt
> sfield=LatLonRPT__location_rpt pt="53.409490 -2.979677" d=25} returns
>
> {
>                 "Id": "HAR/CH1/80763270",
>                 "LatLonRPT__location_rpt": "53.2 -2.916666",
>                 "_dist_": 24.295607
> },
> {
>                 "Id": "HAR/CH42/1918283949",
>                 "LatLonRPT__location_rpt": "53.393239 -3.028859",
>                 "_dist_": 5.7587695
> }
>
> The true distances for these results are 23.67 and 3.73 km and other
> results at a true distance of 17 km aren't returned within the 25 km radius.
>
> The explain has the following
>
> +IntersectsPrefixTreeQuery(IntersectsPrefixTreeQuery(fieldName=LatLonR
> +PT__location_rpt,queryShape=Circle(Pt(x=53.40949,y=-2.979677),
> d=0.2° 25.00km),detailLevel=6,prefixGridScanLevel=7))
>
> Is my set up incorrect in some way or is the
> SpatialRecursivePrefixTreeFieldType not suitable for doing radius
> searches on points in this way?
>
> Thanks in anticipation for any suggestions.
>
> Peter Lancaster.
>
> ________________________________
> This message is confidential and may contain privileged information.
> You should not disclose its contents to any other person. If you are
> not the intended recipient, please notify the sender named above
> immediately. It is expressly declared that this e-mail does not
> constitute nor form part of a contract or unilateral obligation.
> Opinions, conclusions and other information in this message that do
> not relate to the official business of findmypast shall be understood as neither given nor endorsed by it.
> ________________________________
>
> ______________________________________________________________________
> ____
>
> This email has been checked for virus and other malicious content
> prior to leaving our network.
> ______________________________________________________________________
> ____

--
Lucene/Solr Search Committer (PMC), Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com
________________________________

This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of findmypast shall be understood as neither given nor endorsed by it.
________________________________

Re: Geofilt and distance measurement problems using SpatialRecursivePrefixTreeFieldType field type

Posted by David Smiley <da...@gmail.com>.
For latitude and longitude data, I recommend "lat,lon" and never use "x
y".  Perhaps the latter should be an error when geo=true (and inverse when
false) but it isn't.  Yes the documentation could be better!

On Fri, Dec 21, 2018 at 4:31 AM Peter Lancaster <
peter.lancaster@findmypast.com> wrote:

> Hi David,
>
> Ignore my previous reply.
>
> I think you've supplied the answer. Yes we do need to use a space to index
> points in an rpt field, but when we do that the order is flipped from
> Lat,Lon to Lon Lat, so we need to re-index our data. In my defence that is
> far from obvious in the documentation.
>
> Thanks again for your help.
>
> Cheers,
> Peter.
>
> -----Original Message-----
> From: David Smiley [mailto:david.w.smiley@gmail.com]
> Sent: 21 December 2018 04:44
> To: solr-user@lucene.apache.org
> Subject: Re: Geofilt and distance measurement problems using
> SpatialRecursivePrefixTreeFieldType field type
>
> Hi Peter,
>
> Use of an RPT field for distance sorting/boosting is to be avoided where
> possible because it's very inefficient at this specific use-case.  Simply
> use LatLonType for this task, and continue to use RPT for the filter/search
> use-case.
>
> Also I see you putting a space between the coordinates instead of a
> comma...   yet you have geo (latitude & longitude data) so this is a bit
> confusing.  Do "lat,lon".  I think a space will be interpreted as "x y"
> (thus reversed).  Perhaps you've mixed up the coordinates and this
> explains the error?  A quick lookup of your sample coordinates suggests to
> me this is likely the problem.  It's a common mistake.
>
> BTW this:
> maxDistErr="0.2" distanceUnits="kilometers"
> means 200m accuracy (or better).  Is this what you want?  Just checking.
>
> ~ David
>
> On Thu, Dec 13, 2018 at 6:38 AM Peter Lancaster <
> peter.lancaster@findmypast.com> wrote:
>
> > I am currently using Solr 5.5.2 and implementing a GeoSpatial search
> > that returns results within a radius in Km of a specified LatLon.
> > Using a field of type solr.LatLonType and a geofilt query this gives
> > good results but is much slower than our regular queries. Using a bbox
> > query is faster but of course less accurate.
> >
> > I then attempted to use a field of type
> > solr.SpatialRecursivePrefixTreeFieldType to check performance and
> > because I want to be able to do searches within a polygon eventually.
> > The field is defined as follows
> >
> > <fieldType name="location_rpt"
> >  class="solr.SpatialRecursivePrefixTreeFieldType"
> >
> spatialContextFactory="com.spatial4j.core.context.jts.JtsSpatialContextFactory"
> >             geo="true" distErrPct="0.05" maxDistErr="0.2"
> > distanceUnits="kilometers" autoIndex="true"/>
> >
> > <dynamicField name="*__location_rpt" type="location_rpt" indexed="true"
> > stored="true" multiValued="false" omitNorms="true" />
> >
> > I'm just using it to index single points right now. The problem is
> > that the distance calculation is not working correctly. It seems to
> > overstate the distances for differences in longitude.
> >
> > For example a query for
> > &fl=Id,LatLonRPT__location_rpt,_dist_:geodist()&sfield=LatLonRPT__loca
> > tion_rpt&pt=53.409490 -2.979677&query={!geofilt
> > sfield=LatLonRPT__location_rpt pt="53.409490 -2.979677" d=25} returns
> >
> > {
> >                 "Id": "HAR/CH1/80763270",
> >                 "LatLonRPT__location_rpt": "53.2 -2.916666",
> >                 "_dist_": 24.295607
> > },
> > {
> >                 "Id": "HAR/CH42/1918283949",
> >                 "LatLonRPT__location_rpt": "53.393239 -3.028859",
> >                 "_dist_": 5.7587695
> > }
> >
> > The true distances for these results are 23.67 and 3.73 km and other
> > results at a true distance of 17 km aren't returned within the 25 km
> radius.
> >
> > The explain has the following
> >
> > +IntersectsPrefixTreeQuery(IntersectsPrefixTreeQuery(fieldName=LatLonR
> > +PT__location_rpt,queryShape=Circle(Pt(x=53.40949,y=-2.979677),
> > d=0.2° 25.00km),detailLevel=6,prefixGridScanLevel=7))
> >
> > Is my set up incorrect in some way or is the
> > SpatialRecursivePrefixTreeFieldType not suitable for doing radius
> > searches on points in this way?
> >
> > Thanks in anticipation for any suggestions.
> >
> > Peter Lancaster.
> >
> > ________________________________
> > This message is confidential and may contain privileged information.
> > You should not disclose its contents to any other person. If you are
> > not the intended recipient, please notify the sender named above
> > immediately. It is expressly declared that this e-mail does not
> > constitute nor form part of a contract or unilateral obligation.
> > Opinions, conclusions and other information in this message that do
> > not relate to the official business of findmypast shall be understood as
> neither given nor endorsed by it.
> > ________________________________
> >
> > ______________________________________________________________________
> > ____
> >
> > This email has been checked for virus and other malicious content
> > prior to leaving our network.
> > ______________________________________________________________________
> > ____
>
> --
> Lucene/Solr Search Committer (PMC), Developer, Author, Speaker
> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
> http://www.solrenterprisesearchserver.com
> ________________________________
>
> This message is confidential and may contain privileged information. You
> should not disclose its contents to any other person. If you are not the
> intended recipient, please notify the sender named above immediately. It is
> expressly declared that this e-mail does not constitute nor form part of a
> contract or unilateral obligation. Opinions, conclusions and other
> information in this message that do not relate to the official business of
> findmypast shall be understood as neither given nor endorsed by it.
> ________________________________
>
-- 
Lucene/Solr Search Committer (PMC), Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

RE: Geofilt and distance measurement problems using SpatialRecursivePrefixTreeFieldType field type

Posted by Peter Lancaster <pe...@findmypast.com>.
Hi David,

Ignore my previous reply.

I think you've supplied the answer. Yes we do need to use a space to index points in an rpt field, but when we do that the order is flipped from Lat,Lon to Lon Lat, so we need to re-index our data. In my defence that is far from obvious in the documentation.

Thanks again for your help.

Cheers,
Peter.

-----Original Message-----
From: David Smiley [mailto:david.w.smiley@gmail.com]
Sent: 21 December 2018 04:44
To: solr-user@lucene.apache.org
Subject: Re: Geofilt and distance measurement problems using SpatialRecursivePrefixTreeFieldType field type

Hi Peter,

Use of an RPT field for distance sorting/boosting is to be avoided where possible because it's very inefficient at this specific use-case.  Simply use LatLonType for this task, and continue to use RPT for the filter/search use-case.

Also I see you putting a space between the coordinates instead of a
comma...   yet you have geo (latitude & longitude data) so this is a bit
confusing.  Do "lat,lon".  I think a space will be interpreted as "x y"
(thus reversed).  Perhaps you've mixed up the coordinates and this explains the error?  A quick lookup of your sample coordinates suggests to me this is likely the problem.  It's a common mistake.

BTW this:
maxDistErr="0.2" distanceUnits="kilometers"
means 200m accuracy (or better).  Is this what you want?  Just checking.

~ David

On Thu, Dec 13, 2018 at 6:38 AM Peter Lancaster < peter.lancaster@findmypast.com> wrote:

> I am currently using Solr 5.5.2 and implementing a GeoSpatial search
> that returns results within a radius in Km of a specified LatLon.
> Using a field of type solr.LatLonType and a geofilt query this gives
> good results but is much slower than our regular queries. Using a bbox
> query is faster but of course less accurate.
>
> I then attempted to use a field of type
> solr.SpatialRecursivePrefixTreeFieldType to check performance and
> because I want to be able to do searches within a polygon eventually.
> The field is defined as follows
>
> <fieldType name="location_rpt"
>  class="solr.SpatialRecursivePrefixTreeFieldType"
> spatialContextFactory="com.spatial4j.core.context.jts.JtsSpatialContextFactory"
>             geo="true" distErrPct="0.05" maxDistErr="0.2"
> distanceUnits="kilometers" autoIndex="true"/>
>
> <dynamicField name="*__location_rpt" type="location_rpt" indexed="true"
> stored="true" multiValued="false" omitNorms="true" />
>
> I'm just using it to index single points right now. The problem is
> that the distance calculation is not working correctly. It seems to
> overstate the distances for differences in longitude.
>
> For example a query for
> &fl=Id,LatLonRPT__location_rpt,_dist_:geodist()&sfield=LatLonRPT__loca
> tion_rpt&pt=53.409490 -2.979677&query={!geofilt
> sfield=LatLonRPT__location_rpt pt="53.409490 -2.979677" d=25} returns
>
> {
>                 "Id": "HAR/CH1/80763270",
>                 "LatLonRPT__location_rpt": "53.2 -2.916666",
>                 "_dist_": 24.295607
> },
> {
>                 "Id": "HAR/CH42/1918283949",
>                 "LatLonRPT__location_rpt": "53.393239 -3.028859",
>                 "_dist_": 5.7587695
> }
>
> The true distances for these results are 23.67 and 3.73 km and other
> results at a true distance of 17 km aren't returned within the 25 km radius.
>
> The explain has the following
>
> +IntersectsPrefixTreeQuery(IntersectsPrefixTreeQuery(fieldName=LatLonR
> +PT__location_rpt,queryShape=Circle(Pt(x=53.40949,y=-2.979677),
> d=0.2° 25.00km),detailLevel=6,prefixGridScanLevel=7))
>
> Is my set up incorrect in some way or is the
> SpatialRecursivePrefixTreeFieldType not suitable for doing radius
> searches on points in this way?
>
> Thanks in anticipation for any suggestions.
>
> Peter Lancaster.
>
> ________________________________
> This message is confidential and may contain privileged information.
> You should not disclose its contents to any other person. If you are
> not the intended recipient, please notify the sender named above
> immediately. It is expressly declared that this e-mail does not
> constitute nor form part of a contract or unilateral obligation.
> Opinions, conclusions and other information in this message that do
> not relate to the official business of findmypast shall be understood as neither given nor endorsed by it.
> ________________________________
>
> ______________________________________________________________________
> ____
>
> This email has been checked for virus and other malicious content
> prior to leaving our network.
> ______________________________________________________________________
> ____

--
Lucene/Solr Search Committer (PMC), Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com
________________________________

This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of findmypast shall be understood as neither given nor endorsed by it.
________________________________

Re: Geofilt and distance measurement problems using SpatialRecursivePrefixTreeFieldType field type

Posted by David Smiley <da...@gmail.com>.
Hi Peter,

Use of an RPT field for distance sorting/boosting is to be avoided where
possible because it's very inefficient at this specific use-case.  Simply
use LatLonType for this task, and continue to use RPT for the filter/search
use-case.

Also I see you putting a space between the coordinates instead of a
comma...   yet you have geo (latitude & longitude data) so this is a bit
confusing.  Do "lat,lon".  I think a space will be interpreted as "x y"
(thus reversed).  Perhaps you've mixed up the coordinates and this explains
the error?  A quick lookup of your sample coordinates suggests to me this
is likely the problem.  It's a common mistake.

BTW this:
maxDistErr="0.2" distanceUnits="kilometers"
means 200m accuracy (or better).  Is this what you want?  Just checking.

~ David

On Thu, Dec 13, 2018 at 6:38 AM Peter Lancaster <
peter.lancaster@findmypast.com> wrote:

> I am currently using Solr 5.5.2 and implementing a GeoSpatial search that
> returns results within a radius in Km of a specified LatLon. Using a field
> of type solr.LatLonType and a geofilt query this gives good results but is
> much slower than our regular queries. Using a bbox query is faster but of
> course less accurate.
>
> I then attempted to use a field of type
> solr.SpatialRecursivePrefixTreeFieldType to check performance and because I
> want to be able to do searches within a polygon eventually. The field is
> defined as follows
>
> <fieldType name="location_rpt"
>  class="solr.SpatialRecursivePrefixTreeFieldType"
> spatialContextFactory="com.spatial4j.core.context.jts.JtsSpatialContextFactory"
>             geo="true" distErrPct="0.05" maxDistErr="0.2"
> distanceUnits="kilometers" autoIndex="true"/>
>
> <dynamicField name="*__location_rpt" type="location_rpt" indexed="true"
> stored="true" multiValued="false" omitNorms="true" />
>
> I'm just using it to index single points right now. The problem is that
> the distance calculation is not working correctly. It seems to overstate
> the distances for differences in longitude.
>
> For example a query for
> &fl=Id,LatLonRPT__location_rpt,_dist_:geodist()&sfield=LatLonRPT__location_rpt&pt=53.409490
> -2.979677&query={!geofilt sfield=LatLonRPT__location_rpt pt="53.409490
> -2.979677" d=25} returns
>
> {
>                 "Id": "HAR/CH1/80763270",
>                 "LatLonRPT__location_rpt": "53.2 -2.916666",
>                 "_dist_": 24.295607
> },
> {
>                 "Id": "HAR/CH42/1918283949",
>                 "LatLonRPT__location_rpt": "53.393239 -3.028859",
>                 "_dist_": 5.7587695
> }
>
> The true distances for these results are 23.67 and 3.73 km and other
> results at a true distance of 17 km aren't returned within the 25 km radius.
>
> The explain has the following
>
> +IntersectsPrefixTreeQuery(IntersectsPrefixTreeQuery(fieldName=LatLonRPT__location_rpt,queryShape=Circle(Pt(x=53.40949,y=-2.979677),
> d=0.2° 25.00km),detailLevel=6,prefixGridScanLevel=7))
>
> Is my set up incorrect in some way or is the
> SpatialRecursivePrefixTreeFieldType not suitable for doing radius searches
> on points in this way?
>
> Thanks in anticipation for any suggestions.
>
> Peter Lancaster.
>
> ________________________________
> This message is confidential and may contain privileged information. You
> should not disclose its contents to any other person. If you are not the
> intended recipient, please notify the sender named above immediately. It is
> expressly declared that this e-mail does not constitute nor form part of a
> contract or unilateral obligation. Opinions, conclusions and other
> information in this message that do not relate to the official business of
> findmypast shall be understood as neither given nor endorsed by it.
> ________________________________
>
> __________________________________________________________________________
>
> This email has been checked for virus and other malicious content prior to
> leaving our network.
> __________________________________________________________________________

-- 
Lucene/Solr Search Committer (PMC), Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com