You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Beale, Jim (US-KOP)" <Ji...@hibu.com> on 2014/02/12 20:05:29 UTC
RE: Indexing spatial fields into SolrCloud (HTTP)
Hi David,
I finally got back to this again, after getting sidetracked for a couple of weeks.
I implemented things in accordance with my understanding of what you wrote below. Using SolrJ, the code to index the spatial field is as follows,
private void addSpatialField(double lat, double lon, SolrInputDocument document) {
StringBuilder sb = new StringBuilder();
sb.append(lat).append(",").append(lon);
document.addField("location", sb.toString());
}
Using Solr 4.3.1 and spatial4j 0.3, I am getting the following error in the solr logs:
1518436 [qtp1529118084-24] ERROR org.apache.solr.core.SolrCore â org.apache.solr.common.SolrException: com.spatial4j.core.exception.InvalidShapeException: Unable to read: Pt(x=-72.544123,y=41.822225)
at org.apache.solr.schema.AbstractSpatialFieldType.parseShape(AbstractSpatialFieldType.java:144)
at org.apache.solr.schema.AbstractSpatialFieldType.createFields(AbstractSpatialFieldType.java:118)
spatial4j 0.3 is looking for something like POINT but Solr is converting my "lat,long" to Pt(x=-72.544123,y=41.822225).
Version mismatch?
Thanks in advance for your help!
Jim Beale
From: Smiley, David W. [mailto:dsmiley@mitre.org]
Sent: Monday, January 13, 2014 11:30 AM
To: Beale, Jim (US-KOP); solr-user@lucene.apache.org
Subject: Re: Indexing spatial fields into SolrCloud (HTTP)
Hello Jim,
By the way, using GeohashPrefixTree.getMaxLevelsPossible() is usually an extreme choice. Instead you probably want to choose only as many levels needed for your distance tolerance. See SpatialPrefixTreeFactory which you can use outright or borrow the code it uses.
Looking at your code, I see you are coding against Solr directly in Java instead of where most people do this as an HTTP web service. But it's unclear in what context your code runs because you are getting into the guts of things that you normally don't have to do, even if your using SolrJ or writing some sort of UpdateRequestProcessor. Simply configure the field type in schema.xml appropriately, and then to index a point simply give Solr a string for the field in "latitude, longitude" format. I don't know why you are using field.tokenStream(analyzer) for the field value - that is clearly wrong and the cause of the error. I think your confusion more has to do with differences in coding to Lucene versus Solr; this being an actual spatial concern. You referenced "SpatialDemoUpdateProcessorFactory" so I see you have looked at SolrSpatialSandbox on GitHub. That particular URP should get some warnings added to it in the code to suggest that you probably should do what it does. If you look at the solrconfig.xml that configures it, there is a warning as follows:
<!-- spatial
Only needed for an SpatialDemoUpdateProcessorFactory which copies spatial objects from one field
to other spatial fields in object form to avoid redundant/inefficient string to spatial object de-serialization.
-->
Even if you have a similar circumstance, you're code doesn't quite look like this URP. You shouldn't need to reference the SpatialStrategy, for example.
~ David
From: <Beale>, "Jim (US-KOP)" <Ji...@hibu.com>>
Date: Friday, January 10, 2014 at 12:15 PM
To: "solr-user@lucene.apache.org<ma...@lucene.apache.org>" <so...@lucene.apache.org>>
Cc: "Smiley, David W." <ds...@mitre.org>>
Subject: Indexing spatial fields into SolrCloud (HTTP)
I am porting an application from Lucene to Solr which makes use of spatial4j for distance searches. The Lucene version works correctly but I am having a problem getting the Solr version to work in the same way.
Lucene version:
SpatialContext geoSpatialCtx = SpatialContext.GEO;
geoSpatialStrategy = new RecursivePrefixTreeStrategy(new GeohashPrefixTree(
geoSpatialCtx, GeohashPrefixTree.getMaxLevelsPossible()), DocumentFieldNames.LOCATION);
Point point = geoSpatialCtx.makePoint(lon, lat);
for (IndexableField field : geoSpatialStrategy.createIndexableFields(point)) {
document.add(field);
}
//Store the field
document.add(new StoredField(geoSpatialStrategy.getFieldName(), geoSpatialCtx.toString(point)));
Solr version:
Point point = geoSpatialCtx.makePoint(lon, lat);
for (IndexableField field : geoSpatialStrategy.createIndexableFields(point)) {
try {
solrDocument.addField(field.name(), field.tokenStream(analyzer));
} catch (IOException e) {
LOGGER.error("Failed to add geo field to Solr index", e);
}
}
// Store the field
solrDocument.addField(geoSpatialStrategy.getFieldName(), geoSpatialCtx.toString(point));
The server-side error is as follows:
Caused by: com.spatial4j.core.exception.InvalidShapeException: Unable to read: org.apache.lucene.spatial.prefix.PrefixTreeStrategy$CellTokenStr\
eam@0
at com.spatial4j.core.io.ShapeReadWriter.readShape(ShapeReadWriter.java:48)
at com.spatial4j.core.context.SpatialContext.readShape(SpatialContext.java:195)
at org.apache.solr.schema.AbstractSpatialFieldType.parseShape(AbstractSpatialFieldType.java:142)
I've seen David Smiley's sample code, specifically the class, SpatialDemoUpdateProcessorFactory, but I can't say that I was able to benefit from it at all.
What I'm trying to do seems like it should be easy - just to index a point for distance searching - but I'm obviously missing something.
Any ideas?
Thanks,
Jim
The information contained in this email message, including any attachments, is intended solely for use by the individual or entity named above and may be confidential. If the reader of this message is not the intended recipient, you are hereby notified that you must not read, use, disclose, distribute or copy any part of this communication. If you have received this communication in error, please immediately notify me by email and destroy the original message, including any attachments. Thank you.
The information contained in this email message, including any attachments, is intended solely for use by the individual or entity named above and may be confidential. If the reader of this message is not the intended recipient, you are hereby notified that you must not read, use, disclose, distribute or copy any part of this communication. If you have received this communication in error, please immediately notify me by email and destroy the original message, including any attachments. Thank you.
Re: Indexing spatial fields into SolrCloud (HTTP)
Posted by "Smiley, David W." <ds...@mitre.org>.
Your new code should also work, and should be equivalent.
The longer stack trace you have is of the wrapping SolrException which wraps another exception — InvalidShapeException. You should also see the stack trace of InvalidShapeException which should originate out of Spatial4j.
~ David
From: <Beale>, "Jim (US-KOP)" <Ji...@hibu.com>>
Date: Wednesday, February 12, 2014 at 5:21 PM
To: "Smiley, David W." <ds...@mitre.org>>, "solr-user@lucene.apache.org<ma...@lucene.apache.org>" <so...@lucene.apache.org>>
Subject: RE: Indexing spatial fields into SolrCloud (HTTP)
Hi David,
You wrote:
> Perhaps you’ve got some funky UpdateRequestProcessor from experimentation you’ve done that’s parsing then toString’ing it?
No, nothing at all. The update processing is straight out-of-the-box Solr.
> And also, your stack trace should have more to it than what you present here.
I trimmed the stack trace because it seemed like TMI, but here it is for completeness’ sake:
1518436 [qtp1529118084-24] ERROR org.apache.solr.core.SolrCore â org.apache.solr.common.SolrException: com.spatial4j.core.exception.InvalidShapeException: Unable to read: Pt(x=-72.544123,y=41.822225)
at org.apache.solr.schema.AbstractSpatialFieldType.parseShape(AbstractSpatialFieldType.java:144)
at org.apache.solr.schema.AbstractSpatialFieldType.createFields(AbstractSpatialFieldType.java:118)
at org.apache.solr.update.DocumentBuilder.addField(DocumentBuilder.java:186)
at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:257)
at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:73)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:199)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:504)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:640)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:396)
at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:246)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:656)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:359)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
…..
> Your SolrJ code looks totally fine.
Finally, I changed the code to the following,
privatevoid addSpatialLcnFields(double lat, double lon, SolrInputDocument document) {
Point point = geoSpatialCtx.makePoint(lon, lat);
document.addField(geoSpatialStrategy.getFieldName(), geoSpatialCtx.toString(point));
}
and at least it isn’t throwing exceptions now. I’m not sure what is going into the index yet. I’ll have to wait for it to finish. (
Does that code seem correct? I want to avoid the deprecated API but so far I haven’t found any alternatives which work.
Thanks,
Jim Beale
From: Smiley, David W. [mailto:dsmiley@mitre.org]
Sent: Wednesday, February 12, 2014 3:07 PM
To: Beale, Jim (US-KOP); solr-user@lucene.apache.org<ma...@lucene.apache.org>
Subject: Re: Indexing spatial fields into SolrCloud (HTTP)
That’s pretty weird. It appears that somehow a Spatial4j Point class is having it’s toString() called on it (which looks like "Pt(x=-72.544123,y=41.822225)" ) and then Spatial4j is trying to parse this which isn’t in a valid format — the toString is more debug-ability. Your SolrJ code looks totally fine. Perhaps you’ve got some funky UpdateRequestProcessor from experimentation you’ve done that’s parsing then toString’ing it? And also, your stack trace should have more to it than what you present here. It may be on the server-side versus the HTTP error page which can get abbreviated.
~ David
From: <Beale>, "Jim (US-KOP)" <Ji...@hibu.com>>
Date: Wednesday, February 12, 2014 at 2:05 PM
To: "Smiley, David W." <ds...@mitre.org>>, "solr-user@lucene.apache.org<ma...@lucene.apache.org>" <so...@lucene.apache.org>>
Subject: RE: Indexing spatial fields into SolrCloud (HTTP)
Hi David,
I finally got back to this again, after getting sidetracked for a couple of weeks.
I implemented things in accordance with my understanding of what you wrote below. Using SolrJ, the code to index the spatial field is as follows,
privatevoid addSpatialField(double lat, double lon, SolrInputDocument document) {
StringBuilder sb = new StringBuilder();
sb.append(lat).append(",").append(lon);
document.addField("location", sb.toString());
}
Using Solr 4.3.1 and spatial4j 0.3, I am getting the following error in the solr logs:
1518436 [qtp1529118084-24] ERROR org.apache.solr.core.SolrCore â org.apache.solr.common.SolrException: com.spatial4j.core.exception.InvalidShapeException: Unable to read: Pt(x=-72.544123,y=41.822225)
at org.apache.solr.schema.AbstractSpatialFieldType.parseShape(AbstractSpatialFieldType.java:144)
at org.apache.solr.schema.AbstractSpatialFieldType.createFields(AbstractSpatialFieldType.java:118)
spatial4j 0.3 is looking for something like POINT but Solr is converting my “lat,long” to Pt(x=-72.544123,y=41.822225).
Version mismatch?
Thanks in advance for your help!
Jim Beale
From: Smiley, David W. [mailto:dsmiley@mitre.org]
Sent: Monday, January 13, 2014 11:30 AM
To: Beale, Jim (US-KOP); solr-user@lucene.apache.org<ma...@lucene.apache.org>
Subject: Re: Indexing spatial fields into SolrCloud (HTTP)
Hello Jim,
By the way, using GeohashPrefixTree.getMaxLevelsPossible() is usually an extreme choice. Instead you probably want to choose only as many levels needed for your distance tolerance. See SpatialPrefixTreeFactory which you can use outright or borrow the code it uses.
Looking at your code, I see you are coding against Solr directly in Java instead of where most people do this as an HTTP web service. But it’s unclear in what context your code runs because you are getting into the guts of things that you normally don’t have to do, even if your using SolrJ or writing some sort of UpdateRequestProcessor. Simply configure the field type in schema.xml appropriately, and then to index a point simply give Solr a string for the field in “latitude, longitude” format. I don’t know why you are using field.tokenStream(analyzer) for the field value — that is clearly wrong and the cause of the error. I think your confusion more has to do with differences in coding to Lucene versus Solr; this being an actual spatial concern. You referenced “SpatialDemoUpdateProcessorFactory” so I see you have looked at SolrSpatialSandbox on GitHub. That particular URP should get some warnings added to it in the code to suggest that you probably should do what it does. If you look at the solrconfig.xml that configures it, there is a warning as follows:
<!-- spatial
Only needed for an SpatialDemoUpdateProcessorFactory which copies spatial objects from one field
to other spatial fields in object form to avoid redundant/inefficient string to spatial object de-serialization.
-->
Even if you have a similar circumstance, you’re code doesn’t quite look like this URP. You shouldn’t need to reference the SpatialStrategy, for example.
~ David
From: <Beale>, "Jim (US-KOP)" <Ji...@hibu.com>>
Date: Friday, January 10, 2014 at 12:15 PM
To: "solr-user@lucene.apache.org<ma...@lucene.apache.org>" <so...@lucene.apache.org>>
Cc: "Smiley, David W." <ds...@mitre.org>>
Subject: Indexing spatial fields into SolrCloud (HTTP)
I am porting an application from Lucene to Solr which makes use of spatial4j for distance searches. The Lucene version works correctly but I am having a problem getting the Solr version to work in the same way.
Lucene version:
SpatialContext geoSpatialCtx = SpatialContext.GEO;
geoSpatialStrategy = new RecursivePrefixTreeStrategy(new GeohashPrefixTree(
geoSpatialCtx, GeohashPrefixTree.getMaxLevelsPossible()), DocumentFieldNames.LOCATION);
Point point = geoSpatialCtx.makePoint(lon, lat);
for (IndexableField field : geoSpatialStrategy.createIndexableFields(point)) {
document.add(field);
}
//Store the field
document.add(new StoredField(geoSpatialStrategy.getFieldName(), geoSpatialCtx.toString(point)));
Solr version:
Point point = geoSpatialCtx.makePoint(lon, lat);
for (IndexableField field : geoSpatialStrategy.createIndexableFields(point)) {
try {
solrDocument.addField(field.name(), field.tokenStream(analyzer));
} catch (IOException e) {
LOGGER.error("Failed to add geo field to Solr index", e);
}
}
// Store the field
solrDocument.addField(geoSpatialStrategy.getFieldName(), geoSpatialCtx.toString(point));
The server-side error is as follows:
Caused by: com.spatial4j.core.exception.InvalidShapeException: Unable to read: org.apache.lucene.spatial.prefix.PrefixTreeStrategy$CellTokenStr\
eam@0
at com.spatial4j.core.io.ShapeReadWriter.readShape(ShapeReadWriter.java:48)
at com.spatial4j.core.context.SpatialContext.readShape(SpatialContext.java:195)
at org.apache.solr.schema.AbstractSpatialFieldType.parseShape(AbstractSpatialFieldType.java:142)
I’ve seen David Smiley’s sample code, specifically the class, SpatialDemoUpdateProcessorFactory, but I can’t say that I was able to benefit from it at all.
What I’m trying to do seems like it should be easy – just to index a point for distance searching – but I’m obviously missing something.
Any ideas?
Thanks,
Jim
The information contained in this email message, including any attachments, is intended solely for use by the individual or entity named above and may be confidential. If the reader of this message is not the intended recipient, you are hereby notified that you must not read, use, disclose, distribute or copy any part of this communication. If you have received this communication in error, please immediately notify me by email and destroy the original message, including any attachments. Thank you.
The information contained in this email message, including any attachments, is intended solely for use by the individual or entity named above and may be confidential. If the reader of this message is not the intended recipient, you are hereby notified that you must not read, use, disclose, distribute or copy any part of this communication. If you have received this communication in error, please immediately notify me by email and destroy the original message, including any attachments. Thank you.
The information contained in this email message, including any attachments, is intended solely for use by the individual or entity named above and may be confidential. If the reader of this message is not the intended recipient, you are hereby notified that you must not read, use, disclose, distribute or copy any part of this communication. If you have received this communication in error, please immediately notify me by email and destroy the original message, including any attachments. Thank you.
RE: Indexing spatial fields into SolrCloud (HTTP)
Posted by "Beale, Jim (US-KOP)" <Ji...@hibu.com>.
Hi David,
You wrote:
> Perhaps you’ve got some funky UpdateRequestProcessor from experimentation you’ve done that’s parsing then toString’ing it?
No, nothing at all. The update processing is straight out-of-the-box Solr.
> And also, your stack trace should have more to it than what you present here.
I trimmed the stack trace because it seemed like TMI, but here it is for completeness’ sake:
1518436 [qtp1529118084-24] ERROR org.apache.solr.core.SolrCore â org.apache.solr.common.SolrException: com.spatial4j.core.exception.InvalidShapeException: Unable to read: Pt(x=-72.544123,y=41.822225)
at org.apache.solr.schema.AbstractSpatialFieldType.parseShape(AbstractSpatialFieldType.java:144)
at org.apache.solr.schema.AbstractSpatialFieldType.createFields(AbstractSpatialFieldType.java:118)
at org.apache.solr.update.DocumentBuilder.addField(DocumentBuilder.java:186)
at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:257)
at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:73)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:199)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:504)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:640)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:396)
at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:246)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:656)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:359)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
…..
> Your SolrJ code looks totally fine.
Finally, I changed the code to the following,
private void addSpatialLcnFields(double lat, double lon, SolrInputDocument document) {
Point point = geoSpatialCtx.makePoint(lon, lat);
document.addField(geoSpatialStrategy.getFieldName(), geoSpatialCtx.toString(point));
}
and at least it isn’t throwing exceptions now. I’m not sure what is going into the index yet. I’ll have to wait for it to finish. (
Does that code seem correct? I want to avoid the deprecated API but so far I haven’t found any alternatives which work.
Thanks,
Jim Beale
From: Smiley, David W. [mailto:dsmiley@mitre.org]
Sent: Wednesday, February 12, 2014 3:07 PM
To: Beale, Jim (US-KOP); solr-user@lucene.apache.org
Subject: Re: Indexing spatial fields into SolrCloud (HTTP)
That’s pretty weird. It appears that somehow a Spatial4j Point class is having it’s toString() called on it (which looks like "Pt(x=-72.544123,y=41.822225)" ) and then Spatial4j is trying to parse this which isn’t in a valid format — the toString is more debug-ability. Your SolrJ code looks totally fine. Perhaps you’ve got some funky UpdateRequestProcessor from experimentation you’ve done that’s parsing then toString’ing it? And also, your stack trace should have more to it than what you present here. It may be on the server-side versus the HTTP error page which can get abbreviated.
~ David
From: <Beale>, "Jim (US-KOP)" <Ji...@hibu.com>>
Date: Wednesday, February 12, 2014 at 2:05 PM
To: "Smiley, David W." <ds...@mitre.org>>, "solr-user@lucene.apache.org<ma...@lucene.apache.org>" <so...@lucene.apache.org>>
Subject: RE: Indexing spatial fields into SolrCloud (HTTP)
Hi David,
I finally got back to this again, after getting sidetracked for a couple of weeks.
I implemented things in accordance with my understanding of what you wrote below. Using SolrJ, the code to index the spatial field is as follows,
privatevoid addSpatialField(double lat, double lon, SolrInputDocument document) {
StringBuilder sb = new StringBuilder();
sb.append(lat).append(",").append(lon);
document.addField("location", sb.toString());
}
Using Solr 4.3.1 and spatial4j 0.3, I am getting the following error in the solr logs:
1518436 [qtp1529118084-24] ERROR org.apache.solr.core.SolrCore â org.apache.solr.common.SolrException: com.spatial4j.core.exception.InvalidShapeException: Unable to read: Pt(x=-72.544123,y=41.822225)
at org.apache.solr.schema.AbstractSpatialFieldType.parseShape(AbstractSpatialFieldType.java:144)
at org.apache.solr.schema.AbstractSpatialFieldType.createFields(AbstractSpatialFieldType.java:118)
spatial4j 0.3 is looking for something like POINT but Solr is converting my “lat,long” to Pt(x=-72.544123,y=41.822225).
Version mismatch?
Thanks in advance for your help!
Jim Beale
From: Smiley, David W. [mailto:dsmiley@mitre.org]
Sent: Monday, January 13, 2014 11:30 AM
To: Beale, Jim (US-KOP); solr-user@lucene.apache.org<ma...@lucene.apache.org>
Subject: Re: Indexing spatial fields into SolrCloud (HTTP)
Hello Jim,
By the way, using GeohashPrefixTree.getMaxLevelsPossible() is usually an extreme choice. Instead you probably want to choose only as many levels needed for your distance tolerance. See SpatialPrefixTreeFactory which you can use outright or borrow the code it uses.
Looking at your code, I see you are coding against Solr directly in Java instead of where most people do this as an HTTP web service. But it’s unclear in what context your code runs because you are getting into the guts of things that you normally don’t have to do, even if your using SolrJ or writing some sort of UpdateRequestProcessor. Simply configure the field type in schema.xml appropriately, and then to index a point simply give Solr a string for the field in “latitude, longitude” format. I don’t know why you are using field.tokenStream(analyzer) for the field value — that is clearly wrong and the cause of the error. I think your confusion more has to do with differences in coding to Lucene versus Solr; this being an actual spatial concern. You referenced “SpatialDemoUpdateProcessorFactory” so I see you have looked at SolrSpatialSandbox on GitHub. That particular URP should get some warnings added to it in the code to suggest that you probably should do what it does. If you look at the solrconfig.xml that configures it, there is a warning as follows:
<!-- spatial
Only needed for an SpatialDemoUpdateProcessorFactory which copies spatial objects from one field
to other spatial fields in object form to avoid redundant/inefficient string to spatial object de-serialization.
-->
Even if you have a similar circumstance, you’re code doesn’t quite look like this URP. You shouldn’t need to reference the SpatialStrategy, for example.
~ David
From: <Beale>, "Jim (US-KOP)" <Ji...@hibu.com>>
Date: Friday, January 10, 2014 at 12:15 PM
To: "solr-user@lucene.apache.org<ma...@lucene.apache.org>" <so...@lucene.apache.org>>
Cc: "Smiley, David W." <ds...@mitre.org>>
Subject: Indexing spatial fields into SolrCloud (HTTP)
I am porting an application from Lucene to Solr which makes use of spatial4j for distance searches. The Lucene version works correctly but I am having a problem getting the Solr version to work in the same way.
Lucene version:
SpatialContext geoSpatialCtx = SpatialContext.GEO;
geoSpatialStrategy = new RecursivePrefixTreeStrategy(new GeohashPrefixTree(
geoSpatialCtx, GeohashPrefixTree.getMaxLevelsPossible()), DocumentFieldNames.LOCATION);
Point point = geoSpatialCtx.makePoint(lon, lat);
for (IndexableField field : geoSpatialStrategy.createIndexableFields(point)) {
document.add(field);
}
//Store the field
document.add(new StoredField(geoSpatialStrategy.getFieldName(), geoSpatialCtx.toString(point)));
Solr version:
Point point = geoSpatialCtx.makePoint(lon, lat);
for (IndexableField field : geoSpatialStrategy.createIndexableFields(point)) {
try {
solrDocument.addField(field.name(), field.tokenStream(analyzer));
} catch (IOException e) {
LOGGER.error("Failed to add geo field to Solr index", e);
}
}
// Store the field
solrDocument.addField(geoSpatialStrategy.getFieldName(), geoSpatialCtx.toString(point));
The server-side error is as follows:
Caused by: com.spatial4j.core.exception.InvalidShapeException: Unable to read: org.apache.lucene.spatial.prefix.PrefixTreeStrategy$CellTokenStr\
eam@0
at com.spatial4j.core.io.ShapeReadWriter.readShape(ShapeReadWriter.java:48)
at com.spatial4j.core.context.SpatialContext.readShape(SpatialContext.java:195)
at org.apache.solr.schema.AbstractSpatialFieldType.parseShape(AbstractSpatialFieldType.java:142)
I’ve seen David Smiley’s sample code, specifically the class, SpatialDemoUpdateProcessorFactory, but I can’t say that I was able to benefit from it at all.
What I’m trying to do seems like it should be easy – just to index a point for distance searching – but I’m obviously missing something.
Any ideas?
Thanks,
Jim
The information contained in this email message, including any attachments, is intended solely for use by the individual or entity named above and may be confidential. If the reader of this message is not the intended recipient, you are hereby notified that you must not read, use, disclose, distribute or copy any part of this communication. If you have received this communication in error, please immediately notify me by email and destroy the original message, including any attachments. Thank you.
The information contained in this email message, including any attachments, is intended solely for use by the individual or entity named above and may be confidential. If the reader of this message is not the intended recipient, you are hereby notified that you must not read, use, disclose, distribute or copy any part of this communication. If you have received this communication in error, please immediately notify me by email and destroy the original message, including any attachments. Thank you.
The information contained in this email message, including any attachments, is intended solely for use by the individual or entity named above and may be confidential. If the reader of this message is not the intended recipient, you are hereby notified that you must not read, use, disclose, distribute or copy any part of this communication. If you have received this communication in error, please immediately notify me by email and destroy the original message, including any attachments. Thank you.
Re: Indexing spatial fields into SolrCloud (HTTP)
Posted by "Smiley, David W." <ds...@mitre.org>.
That’s pretty weird. It appears that somehow a Spatial4j Point class is having it’s toString() called on it (which looks like "Pt(x=-72.544123,y=41.822225)" ) and then Spatial4j is trying to parse this which isn’t in a valid format — the toString is more debug-ability. Your SolrJ code looks totally fine. Perhaps you’ve got some funky UpdateRequestProcessor from experimentation you’ve done that’s parsing then toString’ing it? And also, your stack trace should have more to it than what you present here. It may be on the server-side versus the HTTP error page which can get abbreviated.
~ David
From: <Beale>, "Jim (US-KOP)" <Ji...@hibu.com>>
Date: Wednesday, February 12, 2014 at 2:05 PM
To: "Smiley, David W." <ds...@mitre.org>>, "solr-user@lucene.apache.org<ma...@lucene.apache.org>" <so...@lucene.apache.org>>
Subject: RE: Indexing spatial fields into SolrCloud (HTTP)
Hi David,
I finally got back to this again, after getting sidetracked for a couple of weeks.
I implemented things in accordance with my understanding of what you wrote below. Using SolrJ, the code to index the spatial field is as follows,
privatevoid addSpatialField(double lat, double lon, SolrInputDocument document) {
StringBuilder sb = new StringBuilder();
sb.append(lat).append(",").append(lon);
document.addField("location", sb.toString());
}
Using Solr 4.3.1 and spatial4j 0.3, I am getting the following error in the solr logs:
1518436 [qtp1529118084-24] ERROR org.apache.solr.core.SolrCore â org.apache.solr.common.SolrException: com.spatial4j.core.exception.InvalidShapeException: Unable to read: Pt(x=-72.544123,y=41.822225)
at org.apache.solr.schema.AbstractSpatialFieldType.parseShape(AbstractSpatialFieldType.java:144)
at org.apache.solr.schema.AbstractSpatialFieldType.createFields(AbstractSpatialFieldType.java:118)
spatial4j 0.3 is looking for something like POINT but Solr is converting my “lat,long” to Pt(x=-72.544123,y=41.822225).
Version mismatch?
Thanks in advance for your help!
Jim Beale
From: Smiley, David W. [mailto:dsmiley@mitre.org]
Sent: Monday, January 13, 2014 11:30 AM
To: Beale, Jim (US-KOP); solr-user@lucene.apache.org<ma...@lucene.apache.org>
Subject: Re: Indexing spatial fields into SolrCloud (HTTP)
Hello Jim,
By the way, using GeohashPrefixTree.getMaxLevelsPossible() is usually an extreme choice. Instead you probably want to choose only as many levels needed for your distance tolerance. See SpatialPrefixTreeFactory which you can use outright or borrow the code it uses.
Looking at your code, I see you are coding against Solr directly in Java instead of where most people do this as an HTTP web service. But it’s unclear in what context your code runs because you are getting into the guts of things that you normally don’t have to do, even if your using SolrJ or writing some sort of UpdateRequestProcessor. Simply configure the field type in schema.xml appropriately, and then to index a point simply give Solr a string for the field in “latitude, longitude” format. I don’t know why you are using field.tokenStream(analyzer) for the field value — that is clearly wrong and the cause of the error. I think your confusion more has to do with differences in coding to Lucene versus Solr; this being an actual spatial concern. You referenced “SpatialDemoUpdateProcessorFactory” so I see you have looked at SolrSpatialSandbox on GitHub. That particular URP should get some warnings added to it in the code to suggest that you probably should do what it does. If you look at the solrconfig.xml that configures it, there is a warning as follows:
<!-- spatial
Only needed for an SpatialDemoUpdateProcessorFactory which copies spatial objects from one field
to other spatial fields in object form to avoid redundant/inefficient string to spatial object de-serialization.
-->
Even if you have a similar circumstance, you’re code doesn’t quite look like this URP. You shouldn’t need to reference the SpatialStrategy, for example.
~ David
From: <Beale>, "Jim (US-KOP)" <Ji...@hibu.com>>
Date: Friday, January 10, 2014 at 12:15 PM
To: "solr-user@lucene.apache.org<ma...@lucene.apache.org>" <so...@lucene.apache.org>>
Cc: "Smiley, David W." <ds...@mitre.org>>
Subject: Indexing spatial fields into SolrCloud (HTTP)
I am porting an application from Lucene to Solr which makes use of spatial4j for distance searches. The Lucene version works correctly but I am having a problem getting the Solr version to work in the same way.
Lucene version:
SpatialContext geoSpatialCtx = SpatialContext.GEO;
geoSpatialStrategy = new RecursivePrefixTreeStrategy(new GeohashPrefixTree(
geoSpatialCtx, GeohashPrefixTree.getMaxLevelsPossible()), DocumentFieldNames.LOCATION);
Point point = geoSpatialCtx.makePoint(lon, lat);
for (IndexableField field : geoSpatialStrategy.createIndexableFields(point)) {
document.add(field);
}
//Store the field
document.add(new StoredField(geoSpatialStrategy.getFieldName(), geoSpatialCtx.toString(point)));
Solr version:
Point point = geoSpatialCtx.makePoint(lon, lat);
for (IndexableField field : geoSpatialStrategy.createIndexableFields(point)) {
try {
solrDocument.addField(field.name(), field.tokenStream(analyzer));
} catch (IOException e) {
LOGGER.error("Failed to add geo field to Solr index", e);
}
}
// Store the field
solrDocument.addField(geoSpatialStrategy.getFieldName(), geoSpatialCtx.toString(point));
The server-side error is as follows:
Caused by: com.spatial4j.core.exception.InvalidShapeException: Unable to read: org.apache.lucene.spatial.prefix.PrefixTreeStrategy$CellTokenStr\
eam@0
at com.spatial4j.core.io.ShapeReadWriter.readShape(ShapeReadWriter.java:48)
at com.spatial4j.core.context.SpatialContext.readShape(SpatialContext.java:195)
at org.apache.solr.schema.AbstractSpatialFieldType.parseShape(AbstractSpatialFieldType.java:142)
I’ve seen David Smiley’s sample code, specifically the class, SpatialDemoUpdateProcessorFactory, but I can’t say that I was able to benefit from it at all.
What I’m trying to do seems like it should be easy – just to index a point for distance searching – but I’m obviously missing something.
Any ideas?
Thanks,
Jim
The information contained in this email message, including any attachments, is intended solely for use by the individual or entity named above and may be confidential. If the reader of this message is not the intended recipient, you are hereby notified that you must not read, use, disclose, distribute or copy any part of this communication. If you have received this communication in error, please immediately notify me by email and destroy the original message, including any attachments. Thank you.
The information contained in this email message, including any attachments, is intended solely for use by the individual or entity named above and may be confidential. If the reader of this message is not the intended recipient, you are hereby notified that you must not read, use, disclose, distribute or copy any part of this communication. If you have received this communication in error, please immediately notify me by email and destroy the original message, including any attachments. Thank you.