You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Chambeda <ch...@gmail.com> on 2012/03/16 18:19:13 UTC
Error while trying to load JSON
I am trying to load a json document that has the following structure:
...
"accessoriesImage": null,
"department": "ET",
"shipping": [
{
"nextDay": 10.19,
"secondDay": 6.45,
"ground": 1.69
}
],
"preowned": false,
"format": "CD",
...
When executing the curl request to store the document in solr I get the
following error:
p>Problem accessing /solr/update/json. Reason:
<pre> invalid key: nextDay [948]</pre></p><hr />/<small>Powered by
Jetty://</small>/<br/>
the JSON is valid, so I am not sure what I need to do to get this to pass.
Any ideas?
--
View this message in context: http://lucene.472066.n3.nabble.com/Error-while-trying-to-load-JSON-tp3832518p3832518.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Error while trying to load JSON
Posted by Chris Hostetter <ho...@fucit.org>.
: Ok, so my issue is that it must be a flat structure. Why isn't the JSON
: parser able to deconstruct the object into a flatter structure for indexing?
: Shouldn't it be able to take any valid JSON structure?
Becuase no one has contributed code to do it?
In general accepting any arbitrary JSON structures (or arbitary xml
structures) and deciding how to flatten that and what the resulting field
names should be is non-trivial.
there would need to be a set of default
mappings/assumptions, and configuration options for people who want to
tweak those default mappings/assumptions.
Your example had...
"shipping": [
{
"nextDay": 10.19,
"secondDay": 6.45,
"ground": 1.69
}
],
...where "shipping" is an array that can evidently contain multiple
sub-maps. should the keys in those sub-maps be mapped to fields like...
shipping_0_nextDay = 10.19
shipping_0_secondDay = 6.45
shipping_0_ground = 1.69
...
shipping_1_nextDay = ...
...or should things like "shipping_nextDay" be a multivalued field
collecting all of the values of the sub-maps in order...
shipping_nextDay = [ 10.19, ... ]
shipping_secondDay = [ 6.45, ... ]
shipping_ground = [ 1.69, ... ]
...your expecations are not neccessarily the same as everyone elses, so
the lowest common denominator is to have the clients decide exactly what
the field = value pairs should be and send them precisely.
-Hoss
Re: Error while trying to load JSON
Posted by Pulkit Singhal <pu...@gmail.com>.
It seems that you are using the bbyopen data. If have made up your mind on
using the JSON data then simply store it in ElasticSearch instead of Solr
as they do take any valid JSON structure. Otherwise, you can download the
xml archive from bbyopen and prepare a schema:
Here are some generic instructions to familiarize you with building schema
given arbitrary data, it should help speed things up, they don't apply
directly to bbyopen data though:
http://pulkitsinghal.blogspot.com/2011/10/import-dynamic-fields-from-xml-into.html
http://pulkitsinghal.blogspot.com/2011/09/import-data-from-amazon-rss-feeds-into.html
Keep in mind, ES also does you a favor by building the right schema
dynamically on the fly as you feed it the JSON data. So it is much easier
to work with.
On Fri, Mar 16, 2012 at 1:26 PM, Erick Erickson <er...@gmail.com>wrote:
> bq: Shouldn't it be able to take any valid JSON structure?
>
> No, that was never the intent. The intent here was just to provide
> a JSON-compatible format for indexing data for those who
> don't like/want to use XML or SolrJ or.... Solr doesn't index arbitrary
> XML either. And I have a hard time imagining what the
> schema.xml file would look like when trying to map
> arbitrary JSON (or XML or....) into fields.
>
> Best
> Erick
>
> On Fri, Mar 16, 2012 at 12:54 PM, Chambeda <ch...@gmail.com> wrote:
> > Ok, so my issue is that it must be a flat structure. Why isn't the JSON
> > parser able to deconstruct the object into a flatter structure for
> indexing?
> > Shouldn't it be able to take any valid JSON structure?
> >
> > --
> > View this message in context:
> http://lucene.472066.n3.nabble.com/Error-while-trying-to-load-JSON-tp3832518p3832611.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Error while trying to load JSON
Posted by Erick Erickson <er...@gmail.com>.
bq: Shouldn't it be able to take any valid JSON structure?
No, that was never the intent. The intent here was just to provide
a JSON-compatible format for indexing data for those who
don't like/want to use XML or SolrJ or.... Solr doesn't index arbitrary
XML either. And I have a hard time imagining what the
schema.xml file would look like when trying to map
arbitrary JSON (or XML or....) into fields.
Best
Erick
On Fri, Mar 16, 2012 at 12:54 PM, Chambeda <ch...@gmail.com> wrote:
> Ok, so my issue is that it must be a flat structure. Why isn't the JSON
> parser able to deconstruct the object into a flatter structure for indexing?
> Shouldn't it be able to take any valid JSON structure?
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Error-while-trying-to-load-JSON-tp3832518p3832611.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Error while trying to load JSON
Posted by Chambeda <ch...@gmail.com>.
Ok, so my issue is that it must be a flat structure. Why isn't the JSON
parser able to deconstruct the object into a flatter structure for indexing?
Shouldn't it be able to take any valid JSON structure?
--
View this message in context: http://lucene.472066.n3.nabble.com/Error-while-trying-to-load-JSON-tp3832518p3832611.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Error while trying to load JSON
Posted by Erick Erickson <er...@gmail.com>.
I don't believe Solr indexes arbitrary JSON, just as it
does not index arbitrary XML. You need the input
to be quite specific to how Solr expects the data,
it's a relatively flat structure. There is an example
in <solr home>/solr/example/exampledocs/books.json that
will give you an idea of the expected format.
Best
Erick
On Fri, Mar 16, 2012 at 12:19 PM, Chambeda <ch...@gmail.com> wrote:
> I am trying to load a json document that has the following structure:
> ...
> "accessoriesImage": null,
> "department": "ET",
> "shipping": [
> {
> "nextDay": 10.19,
> "secondDay": 6.45,
> "ground": 1.69
> }
> ],
> "preowned": false,
> "format": "CD",
> ...
>
> When executing the curl request to store the document in solr I get the
> following error:
>
> p>Problem accessing /solr/update/json. Reason:
> <pre> invalid key: nextDay [948]</pre></p><hr />/<small>Powered by
> Jetty://</small>/<br/>
>
> the JSON is valid, so I am not sure what I need to do to get this to pass.
> Any ideas?
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Error-while-trying-to-load-JSON-tp3832518p3832518.html
> Sent from the Solr - User mailing list archive at Nabble.com.