You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Sunil Varma <su...@gmail.com> on 2020/07/03 11:48:38 UTC
Out of memory errors with Spatial indexing
We are seeing OOM errors when trying to index some spatial data. I believe
the data itself might not be valid but it shouldn't cause the Server to
crash. We see this on both Solr 7.6 and Solr 8. Below is the input that is
causing the error.
{
"id": "bad_data_1",
"spatialwkt_srpt": "LINESTRING (-126.86037681029909 -90.0
1.0000000150474662E30, 73.58164711175415 -90.0 1.0000000150474662E30,
74.52836551959528 -90.0 1.0000000150474662E30, 74.97006811540834 -90.0
1.0000000150474662E30)"
}
Above dynamic field is mapped to field type "location_rpt" (
solr.SpatialRecursivePrefixTreeFieldType).
Any pointers to get around this issue would be highly appreciated.
Thanks!
Re: Out of memory errors with Spatial indexing
Posted by David Smiley <ds...@apache.org>.
I believe you are experiencing this bug: LUCENE-5056
<https://issues.apache.org/jira/browse/LUCENE-5056>
The fix would probably be adjusting code in here
org.apache.lucene.spatial.query.SpatialArgs#calcDistanceFromErrPct
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Mon, Jul 6, 2020 at 5:18 AM Sunil Varma <su...@gmail.com> wrote:
> Hi David
> Thanks for your response. Yes, I noticed that all the data causing issue
> were at the poles. I tried the "RptWithGeometrySpatialField" field type
> definition but get a "Spatial context does not support S2 spatial
> index"error. Setting "spatialContextFactory="Geo3D" I still see the
> original OOM error .
>
> On Sat, 4 Jul 2020 at 05:49, David Smiley <ds...@apache.org> wrote:
>
> > Hi Sunil,
> >
> > Your shape is at a pole, and I'm aware of a bug causing an exponential
> > explosion of needed grid squares when you have polygons super-close to
> the
> > pole. Might you try S2PrefixTree instead? I forget if this would fix it
> > or not by itself. For indexing non-point data, I recommend
> > class="solr.RptWithGeometrySpatialField" which internally is based off a
> > combination of a course grid and storing the original vector geometry for
> > accurate verification:
> > <fieldType name="srptgeom_s2_geo3d"
> > class="solr.RptWithGeometrySpatialField"
> > prefixTree="s2" />
> > The internally coarser grid will lessen the impact of that pole bug.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
> >
> > On Fri, Jul 3, 2020 at 7:48 AM Sunil Varma <su...@gmail.com>
> > wrote:
> >
> > > We are seeing OOM errors when trying to index some spatial data. I
> > believe
> > > the data itself might not be valid but it shouldn't cause the Server to
> > > crash. We see this on both Solr 7.6 and Solr 8. Below is the input that
> > is
> > > causing the error.
> > >
> > > {
> > > "id": "bad_data_1",
> > > "spatialwkt_srpt": "LINESTRING (-126.86037681029909 -90.0
> > > 1.0000000150474662E30, 73.58164711175415 -90.0 1.0000000150474662E30,
> > > 74.52836551959528 -90.0 1.0000000150474662E30, 74.97006811540834 -90.0
> > > 1.0000000150474662E30)"
> > > }
> > >
> > > Above dynamic field is mapped to field type "location_rpt" (
> > > solr.SpatialRecursivePrefixTreeFieldType).
> > >
> > > Any pointers to get around this issue would be highly appreciated.
> > >
> > > Thanks!
> > >
> >
>
Re: Out of memory errors with Spatial indexing
Posted by Sunil Varma <su...@gmail.com>.
Hi David
Thanks for your response. Yes, I noticed that all the data causing issue
were at the poles. I tried the "RptWithGeometrySpatialField" field type
definition but get a "Spatial context does not support S2 spatial
index"error. Setting "spatialContextFactory="Geo3D" I still see the
original OOM error .
On Sat, 4 Jul 2020 at 05:49, David Smiley <ds...@apache.org> wrote:
> Hi Sunil,
>
> Your shape is at a pole, and I'm aware of a bug causing an exponential
> explosion of needed grid squares when you have polygons super-close to the
> pole. Might you try S2PrefixTree instead? I forget if this would fix it
> or not by itself. For indexing non-point data, I recommend
> class="solr.RptWithGeometrySpatialField" which internally is based off a
> combination of a course grid and storing the original vector geometry for
> accurate verification:
> <fieldType name="srptgeom_s2_geo3d"
> class="solr.RptWithGeometrySpatialField"
> prefixTree="s2" />
> The internally coarser grid will lessen the impact of that pole bug.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Fri, Jul 3, 2020 at 7:48 AM Sunil Varma <su...@gmail.com>
> wrote:
>
> > We are seeing OOM errors when trying to index some spatial data. I
> believe
> > the data itself might not be valid but it shouldn't cause the Server to
> > crash. We see this on both Solr 7.6 and Solr 8. Below is the input that
> is
> > causing the error.
> >
> > {
> > "id": "bad_data_1",
> > "spatialwkt_srpt": "LINESTRING (-126.86037681029909 -90.0
> > 1.0000000150474662E30, 73.58164711175415 -90.0 1.0000000150474662E30,
> > 74.52836551959528 -90.0 1.0000000150474662E30, 74.97006811540834 -90.0
> > 1.0000000150474662E30)"
> > }
> >
> > Above dynamic field is mapped to field type "location_rpt" (
> > solr.SpatialRecursivePrefixTreeFieldType).
> >
> > Any pointers to get around this issue would be highly appreciated.
> >
> > Thanks!
> >
>
Re: Out of memory errors with Spatial indexing
Posted by David Smiley <ds...@apache.org>.
Hi Sunil,
Your shape is at a pole, and I'm aware of a bug causing an exponential
explosion of needed grid squares when you have polygons super-close to the
pole. Might you try S2PrefixTree instead? I forget if this would fix it
or not by itself. For indexing non-point data, I recommend
class="solr.RptWithGeometrySpatialField" which internally is based off a
combination of a course grid and storing the original vector geometry for
accurate verification:
<fieldType name="srptgeom_s2_geo3d" class="solr.RptWithGeometrySpatialField"
prefixTree="s2" />
The internally coarser grid will lessen the impact of that pole bug.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Fri, Jul 3, 2020 at 7:48 AM Sunil Varma <su...@gmail.com> wrote:
> We are seeing OOM errors when trying to index some spatial data. I believe
> the data itself might not be valid but it shouldn't cause the Server to
> crash. We see this on both Solr 7.6 and Solr 8. Below is the input that is
> causing the error.
>
> {
> "id": "bad_data_1",
> "spatialwkt_srpt": "LINESTRING (-126.86037681029909 -90.0
> 1.0000000150474662E30, 73.58164711175415 -90.0 1.0000000150474662E30,
> 74.52836551959528 -90.0 1.0000000150474662E30, 74.97006811540834 -90.0
> 1.0000000150474662E30)"
> }
>
> Above dynamic field is mapped to field type "location_rpt" (
> solr.SpatialRecursivePrefixTreeFieldType).
>
> Any pointers to get around this issue would be highly appreciated.
>
> Thanks!
>