You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Colin Freas <cf...@stsci.edu> on 2016/01/19 17:07:36 UTC
Using Solr's spatial functionality for astronomical catalog
Greetings!
I have recently stood up an instance of Solr, indexing a catalog of about 100M records representing points on the celestial sphere. All of the fields are strings, floats, and non-spatial types. I’d like to convert the positional data to an appropriate spatial point data type supported by Solr.
I have a couple of questions about indexing spatial data using Solr, since it seems spatial4j, and the spatial functionality in Solr generally, is more GIS geared. I worry that the measurements of lat/long on the imperfect sphere of the Earth wouldn’t match up with the astronomical right ascension/declination concept of the perfectly spherical celestial sphere used to record the coordinates of our records.
I’m also worried there might be other assumptions built into spatial4j & Solr based on using a real surface vs a virtual one.
Does anyone have experience doing this, or is there perhaps some documentation specific to this use case that anyone might be able to point me to?
Thanks in advance,
Colin
Re: Using Solr's spatial functionality for astronomical catalog
Posted by "david.w.smiley@gmail.com" <da...@gmail.com>.
Ah; I saw that. I'm glad you figured it out. Yes, you needed the SQL
alias. I'm kinda surprised you didn't get an error about a field by the
name of your expression not existing... but maybe you have a catch-all
dynamic field or maybe you're in data-driven mode. In either case, I'd
expect a quick select of your data would show those fields. And your first
query wasn't a spatial query; there is no spatial=true parameter. You got
it right the second time.
On Tue, Feb 16, 2016 at 4:03 AM Colin Freas <cf...@stsci.edu> wrote:
>
> Looks like the only issue was that I did not have an alias for SourceRpt
> field in the SQL.
>
> With that in place, everything seems to work more or less as expected.
> SourceRpt shows up where it should.
>
> Queries like
>
> http://localhost:8983/solr/spatial/select?q=*:*&fq={!geofilt%20sfield=Sour
> <http://localhost:8983/solr/spatial/select?q=*:*&fq=%7B!geofilt%20sfield=Sour>
> ceRpt}&pt=0,0&d=5
>
>
>
> ... return appropriate subsets.
>
> Doh,
> Colin
>
> On 2/16/16, 3:31 AM, "Colin Freas" <cf...@stsci.edu> wrote:
>
> >
> >David, thanks for getting back to me. SpatialRecursivePrefixTreeFieldType
> >seems to be what I need, and the default search seems appropriate. This
> >is for entries in an astronomical catalog, so great circle distances on a
> >perfect sphere is what I¹m after.
> >
> >I am having a bit of difficulty though.
> >
> >Having gotten records importing via our database into a schema on both a
> >stand-along Solr instance and in a SolrCloud cluster, I¹ve moved on to
> >³spatializing² the appropriate fields, and everything looks like it¹s
> >working, in that there are no errors thrown. But when I try what I think
> >is valid spatial query, it doesn¹t work.
> >
> >Here¹s what I¹m doing. Pertinent bits from my schema:
> >
> > ...
> > <field name="SourceRpt" type="rpt" indexed="true"
> stored="false"
> >required="false" multiValued="false" />
> > <uniqueKey>CatID</uniqueKey>
> > </fields>
> > <types>
> > <fieldType name="rpt"
> class="solr.SpatialRecursivePrefixTreeFieldType"
> > geo="true"
> > distanceUnits="degrees" />
> > ...
> > </types>
> >
> >In my db-config.xml, I¹ve got this sql:
> > <entity name="observation" query="SELECT CatID, MatchRA, MatchDec,
> >SourceRA, SourceDec, '''' + ltrim(rtrim(Str(SourceRA,25,16))) + ',' +
> >ltrim(rtrim(Str(SourceDec,25,16))) + '''' FROM dbo.Catalog" />
> >
> >
> >When I run a data import through Solr¹s admin gui and look at the verbose
> >debug output, something seems off. Top of the output is this:
> > {
> > "responseHeader": {
> > "status": 0,
> > "QTime": 75
> > },
> > "initArgs": [
> > "defaults",
> > [
> > "config",
> > "hsc-db-config.xml"
> > ]
> > ],
> > "command": "full-import",
> > "mode": "debug",
> > "documents": [
> > {
> > "MatchDec": [
> > -0.673125699999921
> > ],
> > "SourceDec": [
> > -0.673125699999921
> > ],
> > "MatchRA": [
> > 0.5681586795334927
> > ],
> > "SourceRA": [
> > 0.5681586795334927
> > ],
> > "CatID": [
> > 25558943
> > ]
> > },
> >
> >
> >There¹s no SourceRpt field there. But in verbose output of what¹s
> >returned from the query, the SourceRpt field seems to be correctly put
> >together:
> >
> > "verbose-output": [
> > "entity:observation",
> > [
> > "document#1",
> > [
> > "query",
> > "SELECT CatID, MatchRA, MatchDec, SourceRA, SourceDec, '''' +
> >ltrim(rtrim(Str(SourceRA,25,16))) + ',' +
> >ltrim(rtrim(Str(SourceDec,25,16))) + '''' FROM xcat.BestCatalog",
> > "time-taken",
> > "0:0:0.22",
> > null,
> > "----------- row #1-------------",
> > "",
> > "'0.5681586795334928,-0.6731256999999210'",
> > "MatchDec",
> > -0.673125699999921,
> > "SourceDec",
> > -0.673125699999921,
> > "MatchRA",
> > 0.5681586795334927,
> > "SourceRA",
> > 0.5681586795334927,
> > "CatID",
> > 25558943,
> > null,
> > "---------------------------------------------"
> > ]
> >
> >
> >
> >I try a spatial search like this:
> >
> http://localhost:8983/solr/spatial/select?q=*%3A*&wt=json&indent=true&spa
> >t
> >ial=true&pt=3%2C3&sfield=SourceRpt&d=0.0001
> >
> >
> >... And I get back all (10) records in the core, when I would expect 0,
> >given the very small distance I supply to a point well away from any of
> >the records.
> >
> >I¹m not sure what¹s going on. I don¹t know if this is a simple Solr
> >config error I¹m missing, or if there¹s some spatial magic I¹m unaware of.
> >
> >
> >Any thoughts appreciated.
> >
> >-Colin
> >
> >
> >On 1/20/16, 9:34 PM, "david.w.smiley@gmail.com" <david.w.smiley@gmail.com
> >
> >wrote:
> >
> >>Hello Colin,
> >>
> >>If the spatial field you use is the SpatialRecursivePrefixTreeFieldType
> >>one
> >>(RPT for short) with geo="true" then the circle shape (i.e. point-radius
> >>filter) implied by the geofilt Solr QParser is on a sphere. That is, it
> >>uses the "great circle" distance computed using the Haversine formula by
> >>default, though it can be configured to use the Law of Cosines formula or
> >>Vincenty (spherical version) formula if you so choose. Using geodist()
> >>for
> >>spatial distance sorting/boosting also uses this. If you use LatLonType
> >>then geofilt & geodist() use Haversine too.
> >>
> >>If you use polygons or line strings, then it's *not* using a spherical
> >>model; it's using a Euclidean (flat) model on plate carrée. I am
> >>currently
> >>working on adapting the Spatial4j library to work with Lucene's Geo3D
> >>(aka
> >>spatial 3d) which has both a spherical model and an ellipsoidal model,
> >>which can be configured with the characteristics specified by WGS84. If
> >>you are super-eager to get this yourself without waiting, then you could
> >>write a Solr QParser that constructs a Geo3dShape wrapping a Geo3D
> >>GeoShape
> >>object constructed from query parameters. You might alternatively try
> >>and
> >>use Geo3DPointField on Lucene 6 trunk.
> >>
> >>~ David
> >>
> >>On Tue, Jan 19, 2016 at 11:07 AM Colin Freas <cf...@stsci.edu> wrote:
> >>
> >>>
> >>> Greetings!
> >>>
> >>> I have recently stood up an instance of Solr, indexing a catalog of
> >>>about
> >>> 100M records representing points on the celestial sphere. All of the
> >>> fields are strings, floats, and non-spatial types. I¹d like to convert
> >>>the
> >>> positional data to an appropriate spatial point data type supported by
> >>>Solr.
> >>>
> >>> I have a couple of questions about indexing spatial data using Solr,
> >>>since
> >>> it seems spatial4j, and the spatial functionality in Solr generally, is
> >>> more GIS geared. I worry that the measurements of lat/long on the
> >>> imperfect sphere of the Earth wouldn¹t match up with the astronomical
> >>>right
> >>> ascension/declination concept of the perfectly spherical celestial
> >>>sphere
> >>> used to record the coordinates of our records.
> >>>
> >>> I¹m also worried there might be other assumptions built into spatial4j
> >>>&
> >>> Solr based on using a real surface vs a virtual one.
> >>>
> >>> Does anyone have experience doing this, or is there perhaps some
> >>> documentation specific to this use case that anyone might be able to
> >>>point
> >>> me to?
> >>>
> >>> Thanks in advance,
> >>> Colin
> >>>
> >>--
> >>Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> >>LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
> >>http://www.solrenterprisesearchserver.com
> >
>
> --
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com
Re: Using Solr's spatial functionality for astronomical catalog
Posted by Colin Freas <cf...@stsci.edu>.
Looks like the only issue was that I did not have an alias for SourceRpt
field in the SQL.
With that in place, everything seems to work more or less as expected.
SourceRpt shows up where it should.
Queries like
http://localhost:8983/solr/spatial/select?q=*:*&fq={!geofilt%20sfield=Sour
ceRpt}&pt=0,0&d=5
... return appropriate subsets.
Doh,
Colin
On 2/16/16, 3:31 AM, "Colin Freas" <cf...@stsci.edu> wrote:
>
>David, thanks for getting back to me. SpatialRecursivePrefixTreeFieldType
>seems to be what I need, and the default search seems appropriate. This
>is for entries in an astronomical catalog, so great circle distances on a
>perfect sphere is what I¹m after.
>
>I am having a bit of difficulty though.
>
>Having gotten records importing via our database into a schema on both a
>stand-along Solr instance and in a SolrCloud cluster, I¹ve moved on to
>³spatializing² the appropriate fields, and everything looks like it¹s
>working, in that there are no errors thrown. But when I try what I think
>is valid spatial query, it doesn¹t work.
>
>Here¹s what I¹m doing. Pertinent bits from my schema:
>
> ...
> <field name="SourceRpt" type="rpt" indexed="true" stored="false"
>required="false" multiValued="false" />
> <uniqueKey>CatID</uniqueKey>
> </fields>
> <types>
> <fieldType name="rpt" class="solr.SpatialRecursivePrefixTreeFieldType"
> geo="true"
> distanceUnits="degrees" />
> ...
> </types>
>
>In my db-config.xml, I¹ve got this sql:
> <entity name="observation" query="SELECT CatID, MatchRA, MatchDec,
>SourceRA, SourceDec, '''' + ltrim(rtrim(Str(SourceRA,25,16))) + ',' +
>ltrim(rtrim(Str(SourceDec,25,16))) + '''' FROM dbo.Catalog" />
>
>
>When I run a data import through Solr¹s admin gui and look at the verbose
>debug output, something seems off. Top of the output is this:
> {
> "responseHeader": {
> "status": 0,
> "QTime": 75
> },
> "initArgs": [
> "defaults",
> [
> "config",
> "hsc-db-config.xml"
> ]
> ],
> "command": "full-import",
> "mode": "debug",
> "documents": [
> {
> "MatchDec": [
> -0.673125699999921
> ],
> "SourceDec": [
> -0.673125699999921
> ],
> "MatchRA": [
> 0.5681586795334927
> ],
> "SourceRA": [
> 0.5681586795334927
> ],
> "CatID": [
> 25558943
> ]
> },
>
>
>There¹s no SourceRpt field there. But in verbose output of what¹s
>returned from the query, the SourceRpt field seems to be correctly put
>together:
>
> "verbose-output": [
> "entity:observation",
> [
> "document#1",
> [
> "query",
> "SELECT CatID, MatchRA, MatchDec, SourceRA, SourceDec, '''' +
>ltrim(rtrim(Str(SourceRA,25,16))) + ',' +
>ltrim(rtrim(Str(SourceDec,25,16))) + '''' FROM xcat.BestCatalog",
> "time-taken",
> "0:0:0.22",
> null,
> "----------- row #1-------------",
> "",
> "'0.5681586795334928,-0.6731256999999210'",
> "MatchDec",
> -0.673125699999921,
> "SourceDec",
> -0.673125699999921,
> "MatchRA",
> 0.5681586795334927,
> "SourceRA",
> 0.5681586795334927,
> "CatID",
> 25558943,
> null,
> "---------------------------------------------"
> ]
>
>
>
>I try a spatial search like this:
> http://localhost:8983/solr/spatial/select?q=*%3A*&wt=json&indent=true&spa
>t
>ial=true&pt=3%2C3&sfield=SourceRpt&d=0.0001
>
>
>... And I get back all (10) records in the core, when I would expect 0,
>given the very small distance I supply to a point well away from any of
>the records.
>
>I¹m not sure what¹s going on. I don¹t know if this is a simple Solr
>config error I¹m missing, or if there¹s some spatial magic I¹m unaware of.
>
>
>Any thoughts appreciated.
>
>-Colin
>
>
>On 1/20/16, 9:34 PM, "david.w.smiley@gmail.com" <da...@gmail.com>
>wrote:
>
>>Hello Colin,
>>
>>If the spatial field you use is the SpatialRecursivePrefixTreeFieldType
>>one
>>(RPT for short) with geo="true" then the circle shape (i.e. point-radius
>>filter) implied by the geofilt Solr QParser is on a sphere. That is, it
>>uses the "great circle" distance computed using the Haversine formula by
>>default, though it can be configured to use the Law of Cosines formula or
>>Vincenty (spherical version) formula if you so choose. Using geodist()
>>for
>>spatial distance sorting/boosting also uses this. If you use LatLonType
>>then geofilt & geodist() use Haversine too.
>>
>>If you use polygons or line strings, then it's *not* using a spherical
>>model; it's using a Euclidean (flat) model on plate carrée. I am
>>currently
>>working on adapting the Spatial4j library to work with Lucene's Geo3D
>>(aka
>>spatial 3d) which has both a spherical model and an ellipsoidal model,
>>which can be configured with the characteristics specified by WGS84. If
>>you are super-eager to get this yourself without waiting, then you could
>>write a Solr QParser that constructs a Geo3dShape wrapping a Geo3D
>>GeoShape
>>object constructed from query parameters. You might alternatively try
>>and
>>use Geo3DPointField on Lucene 6 trunk.
>>
>>~ David
>>
>>On Tue, Jan 19, 2016 at 11:07 AM Colin Freas <cf...@stsci.edu> wrote:
>>
>>>
>>> Greetings!
>>>
>>> I have recently stood up an instance of Solr, indexing a catalog of
>>>about
>>> 100M records representing points on the celestial sphere. All of the
>>> fields are strings, floats, and non-spatial types. I¹d like to convert
>>>the
>>> positional data to an appropriate spatial point data type supported by
>>>Solr.
>>>
>>> I have a couple of questions about indexing spatial data using Solr,
>>>since
>>> it seems spatial4j, and the spatial functionality in Solr generally, is
>>> more GIS geared. I worry that the measurements of lat/long on the
>>> imperfect sphere of the Earth wouldn¹t match up with the astronomical
>>>right
>>> ascension/declination concept of the perfectly spherical celestial
>>>sphere
>>> used to record the coordinates of our records.
>>>
>>> I¹m also worried there might be other assumptions built into spatial4j
>>>&
>>> Solr based on using a real surface vs a virtual one.
>>>
>>> Does anyone have experience doing this, or is there perhaps some
>>> documentation specific to this use case that anyone might be able to
>>>point
>>> me to?
>>>
>>> Thanks in advance,
>>> Colin
>>>
>>--
>>Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
>>LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
>>http://www.solrenterprisesearchserver.com
>
Re: Using Solr's spatial functionality for astronomical catalog
Posted by Colin Freas <cf...@stsci.edu>.
David, thanks for getting back to me. SpatialRecursivePrefixTreeFieldType
seems to be what I need, and the default search seems appropriate. This
is for entries in an astronomical catalog, so great circle distances on a
perfect sphere is what I¹m after.
I am having a bit of difficulty though.
Having gotten records importing via our database into a schema on both a
stand-along Solr instance and in a SolrCloud cluster, I¹ve moved on to
³spatializing² the appropriate fields, and everything looks like it¹s
working, in that there are no errors thrown. But when I try what I think
is valid spatial query, it doesn¹t work.
Here¹s what I¹m doing. Pertinent bits from my schema:
...
<field name="SourceRpt" type="rpt" indexed="true" stored="false"
required="false" multiValued="false" />
<uniqueKey>CatID</uniqueKey>
</fields>
<types>
<fieldType name="rpt" class="solr.SpatialRecursivePrefixTreeFieldType"
geo="true"
distanceUnits="degrees" />
...
</types>
In my db-config.xml, I¹ve got this sql:
<entity name="observation" query="SELECT CatID, MatchRA, MatchDec,
SourceRA, SourceDec, '''' + ltrim(rtrim(Str(SourceRA,25,16))) + ',' +
ltrim(rtrim(Str(SourceDec,25,16))) + '''' FROM dbo.Catalog" />
When I run a data import through Solr¹s admin gui and look at the verbose
debug output, something seems off. Top of the output is this:
{
"responseHeader": {
"status": 0,
"QTime": 75
},
"initArgs": [
"defaults",
[
"config",
"hsc-db-config.xml"
]
],
"command": "full-import",
"mode": "debug",
"documents": [
{
"MatchDec": [
-0.673125699999921
],
"SourceDec": [
-0.673125699999921
],
"MatchRA": [
0.5681586795334927
],
"SourceRA": [
0.5681586795334927
],
"CatID": [
25558943
]
},
There¹s no SourceRpt field there. But in verbose output of what¹s
returned from the query, the SourceRpt field seems to be correctly put
together:
"verbose-output": [
"entity:observation",
[
"document#1",
[
"query",
"SELECT CatID, MatchRA, MatchDec, SourceRA, SourceDec, '''' +
ltrim(rtrim(Str(SourceRA,25,16))) + ',' +
ltrim(rtrim(Str(SourceDec,25,16))) + '''' FROM xcat.BestCatalog",
"time-taken",
"0:0:0.22",
null,
"----------- row #1-------------",
"",
"'0.5681586795334928,-0.6731256999999210'",
"MatchDec",
-0.673125699999921,
"SourceDec",
-0.673125699999921,
"MatchRA",
0.5681586795334927,
"SourceRA",
0.5681586795334927,
"CatID",
25558943,
null,
"---------------------------------------------"
]
I try a spatial search like this:
http://localhost:8983/solr/spatial/select?q=*%3A*&wt=json&indent=true&spat
ial=true&pt=3%2C3&sfield=SourceRpt&d=0.0001
... And I get back all (10) records in the core, when I would expect 0,
given the very small distance I supply to a point well away from any of
the records.
I¹m not sure what¹s going on. I don¹t know if this is a simple Solr
config error I¹m missing, or if there¹s some spatial magic I¹m unaware of.
Any thoughts appreciated.
-Colin
On 1/20/16, 9:34 PM, "david.w.smiley@gmail.com" <da...@gmail.com>
wrote:
>Hello Colin,
>
>If the spatial field you use is the SpatialRecursivePrefixTreeFieldType
>one
>(RPT for short) with geo="true" then the circle shape (i.e. point-radius
>filter) implied by the geofilt Solr QParser is on a sphere. That is, it
>uses the "great circle" distance computed using the Haversine formula by
>default, though it can be configured to use the Law of Cosines formula or
>Vincenty (spherical version) formula if you so choose. Using geodist()
>for
>spatial distance sorting/boosting also uses this. If you use LatLonType
>then geofilt & geodist() use Haversine too.
>
>If you use polygons or line strings, then it's *not* using a spherical
>model; it's using a Euclidean (flat) model on plate carrée. I am
>currently
>working on adapting the Spatial4j library to work with Lucene's Geo3D (aka
>spatial 3d) which has both a spherical model and an ellipsoidal model,
>which can be configured with the characteristics specified by WGS84. If
>you are super-eager to get this yourself without waiting, then you could
>write a Solr QParser that constructs a Geo3dShape wrapping a Geo3D
>GeoShape
>object constructed from query parameters. You might alternatively try and
>use Geo3DPointField on Lucene 6 trunk.
>
>~ David
>
>On Tue, Jan 19, 2016 at 11:07 AM Colin Freas <cf...@stsci.edu> wrote:
>
>>
>> Greetings!
>>
>> I have recently stood up an instance of Solr, indexing a catalog of
>>about
>> 100M records representing points on the celestial sphere. All of the
>> fields are strings, floats, and non-spatial types. I¹d like to convert
>>the
>> positional data to an appropriate spatial point data type supported by
>>Solr.
>>
>> I have a couple of questions about indexing spatial data using Solr,
>>since
>> it seems spatial4j, and the spatial functionality in Solr generally, is
>> more GIS geared. I worry that the measurements of lat/long on the
>> imperfect sphere of the Earth wouldn¹t match up with the astronomical
>>right
>> ascension/declination concept of the perfectly spherical celestial
>>sphere
>> used to record the coordinates of our records.
>>
>> I¹m also worried there might be other assumptions built into spatial4j &
>> Solr based on using a real surface vs a virtual one.
>>
>> Does anyone have experience doing this, or is there perhaps some
>> documentation specific to this use case that anyone might be able to
>>point
>> me to?
>>
>> Thanks in advance,
>> Colin
>>
>--
>Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
>LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
>http://www.solrenterprisesearchserver.com
Re: Using Solr's spatial functionality for astronomical catalog
Posted by "david.w.smiley@gmail.com" <da...@gmail.com>.
Hello Colin,
If the spatial field you use is the SpatialRecursivePrefixTreeFieldType one
(RPT for short) with geo="true" then the circle shape (i.e. point-radius
filter) implied by the geofilt Solr QParser is on a sphere. That is, it
uses the "great circle" distance computed using the Haversine formula by
default, though it can be configured to use the Law of Cosines formula or
Vincenty (spherical version) formula if you so choose. Using geodist() for
spatial distance sorting/boosting also uses this. If you use LatLonType
then geofilt & geodist() use Haversine too.
If you use polygons or line strings, then it's *not* using a spherical
model; it's using a Euclidean (flat) model on plate carrée. I am currently
working on adapting the Spatial4j library to work with Lucene's Geo3D (aka
spatial 3d) which has both a spherical model and an ellipsoidal model,
which can be configured with the characteristics specified by WGS84. If
you are super-eager to get this yourself without waiting, then you could
write a Solr QParser that constructs a Geo3dShape wrapping a Geo3D GeoShape
object constructed from query parameters. You might alternatively try and
use Geo3DPointField on Lucene 6 trunk.
~ David
On Tue, Jan 19, 2016 at 11:07 AM Colin Freas <cf...@stsci.edu> wrote:
>
> Greetings!
>
> I have recently stood up an instance of Solr, indexing a catalog of about
> 100M records representing points on the celestial sphere. All of the
> fields are strings, floats, and non-spatial types. I’d like to convert the
> positional data to an appropriate spatial point data type supported by Solr.
>
> I have a couple of questions about indexing spatial data using Solr, since
> it seems spatial4j, and the spatial functionality in Solr generally, is
> more GIS geared. I worry that the measurements of lat/long on the
> imperfect sphere of the Earth wouldn’t match up with the astronomical right
> ascension/declination concept of the perfectly spherical celestial sphere
> used to record the coordinates of our records.
>
> I’m also worried there might be other assumptions built into spatial4j &
> Solr based on using a real surface vs a virtual one.
>
> Does anyone have experience doing this, or is there perhaps some
> documentation specific to this use case that anyone might be able to point
> me to?
>
> Thanks in advance,
> Colin
>
--
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com