You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Doug Steigerwald <ds...@mcclatchyinteractive.com> on 2008/03/05 15:29:10 UTC
JSONRequestWriter
We're using localsolr and the RubyResponseWriter. When we do a request with the localsolr component
in our requestHandler we're seeing issues with the display of a multivalued field when it only has
one value.
'class'=>['showtime']'showtime', <------
'genre'=>['Drama',
'Suspsense/Triller'],
With no localsolr component it works fine.
Looks like the issue is with the JSONRequestWriter.writeSolrDocument(). Here's the small patch for
it that seems to fix it.
Index: src/java/org/apache/solr/request/JSONResponseWriter.java
===================================================================
--- src/java/org/apache/solr/request/JSONResponseWriter.java (revision 614955)
+++ src/java/org/apache/solr/request/JSONResponseWriter.java (working copy)
@@ -416,7 +416,7 @@
writeVal(fname, val);
writeArrayCloser();
}
- writeVal(fname, val);
+ else writeVal(fname, val);
}
if (pseudoFields !=null && pseudoFields.size()>0) {
We're running solr trunk r614955 (Jan 23rd), and r75 of localsolr.
Result snippet with the patch:
'class'=>['showtime'],
'genre'=>['Drama',
'Suspsense/Triller'],
Has anyone come across an issue like this? Is this fixed in a newer build of Solr? It looks like
we'd still need this patch even in a build of the solr trunk from yesterday, but maybe not.
--
Doug Steigerwald
Software Developer
McClatchy Interactive
dsteigerwald@mcclatchyinteractive.com
919.861.1287
Re: JSONRequestWriter
Posted by Doug Steigerwald <ds...@mcclatchyinteractive.com>.
Sweet. Thanks.
Doug
Yonik Seeley wrote:
> Thanks Doug, I just checked in your fix.
> This was a recent bug... writing of SolrDocument was recently added
> and is not touched by normal code paths, except for distributed
> search.
>
> -Yonik
>
> On Wed, Mar 5, 2008 at 9:29 AM, Doug Steigerwald
> <ds...@mcclatchyinteractive.com> wrote:
>> We're using localsolr and the RubyResponseWriter. When we do a request with the localsolr component
>> in our requestHandler we're seeing issues with the display of a multivalued field when it only has
>> one value.
>>
>> 'class'=>['showtime']'showtime', <------
>> 'genre'=>['Drama',
>> 'Suspsense/Triller'],
>>
>> With no localsolr component it works fine.
>>
>> Looks like the issue is with the JSONRequestWriter.writeSolrDocument(). Here's the small patch for
>> it that seems to fix it.
>>
>> Index: src/java/org/apache/solr/request/JSONResponseWriter.java
>> ===================================================================
>> --- src/java/org/apache/solr/request/JSONResponseWriter.java (revision 614955)
>> +++ src/java/org/apache/solr/request/JSONResponseWriter.java (working copy)
>> @@ -416,7 +416,7 @@
>> writeVal(fname, val);
>> writeArrayCloser();
>> }
>> - writeVal(fname, val);
>> + else writeVal(fname, val);
>> }
>>
>> if (pseudoFields !=null && pseudoFields.size()>0) {
>>
>>
>> We're running solr trunk r614955 (Jan 23rd), and r75 of localsolr.
>>
>> Result snippet with the patch:
>>
>> 'class'=>['showtime'],
>> 'genre'=>['Drama',
>> 'Suspsense/Triller'],
>>
>> Has anyone come across an issue like this? Is this fixed in a newer build of Solr? It looks like
>> we'd still need this patch even in a build of the solr trunk from yesterday, but maybe not.
>>
>> --
>> Doug Steigerwald
>> Software Developer
>> McClatchy Interactive
>> dsteigerwald@mcclatchyinteractive.com
>> 919.861.1287
>>
Re: JSONRequestWriter
Posted by Yonik Seeley <yo...@apache.org>.
I think the container is the issue (the type of rsp).
Here is the Javadoc for SimpleOrderedMap:
/** <code>SimpleOrderedMap</code> is a {@link NamedList} where access
by key is more
* important than maintaining order when it comes to representing the
* held data in other forms, as ResponseWriters normally do.
* It's normally not a good idea to repeat keys or use null keys, but this
* is not enforced. If key uniqueness enforcement is desired, use a
regular {@link Map}.
* <p>
* For example, a JSON response writer may choose to write a SimpleOrderedMap
* as {"foo":10,"bar":20} and may choose to write a NamedList as
* ["foo",10,"bar",20]. An XML response writer may choose to render both
* the same way.
* </p>
* <p>
* This class does not provide efficient lookup by key, it's main purpose is
* to hold data to be serialized. It aims to minimize overhead and to be
* efficient at adding new elements.
* </p>
*/
-Yonik
On Wed, Mar 5, 2008 at 1:56 PM, Doug Steigerwald
<ds...@mcclatchyinteractive.com> wrote:
> Looks like it's only happening when we use the LocalSolrQueryComponent from localsolr.
>
> rsp.add("response", sdoclist);
>
> sdoclist is a SolrDocumentList. Could that be causing an issue instead of it being just a DocList?
>
> Doug
>
>
>
> Yonik Seeley wrote:
> > The output you showed is indeed incorrect, but I can't reproduce that
> > with stock solr.
> > Here is a example of what I get:
> >
> > {
> > 'responseHeader'=>{
> > 'status'=>0,
> > 'QTime'=>16,
> > 'params'=>{
> > 'wt'=>'ruby',
> > 'indent'=>'true',
> > 'q'=>'*:*',
> > 'facet'=>'true',
> > 'highlight'=>'true'}},
> > 'response'=>{'numFound'=>0,'start'=>0,'docs'=>[]
> > },
> > 'facet_counts'=>{
> > 'facet_queries'=>{},
> > 'facet_fields'=>{},
> > 'facet_dates'=>{}}}
> >
> >
> > -Yonik
> >
> > On Wed, Mar 5, 2008 at 12:00 PM, Doug Steigerwald
> > <ds...@mcclatchyinteractive.com> wrote:
> >> Sure.
> >>
> >> The default (json.nl=flat):
> >>
> >> 'response',{'numFound'=>41,'start'=>0,............
> >>
> >> Adding json.nl=map makes output correct:
> >>
> >> 'response'=>{'numFound'=>41,'start'=>0,............
> >>
> >> This also changes facet output (which was evaluating fine):
> >>
> >> FLAT:
> >>
> >> 'facet_counts',{
> >> 'facet_queries'=>{},
> >> 'facet_fields'=>{
> >> 'movies_movie_genre_facet'=>[
> >> 'Drama',22,
> >> 'Action/Adventure',11,
> >> 'Comedy',11,
> >> 'Suspense/Thriller',11,
> >> 'SciFi/Fantasy',5,
> >> 'Animation',4,
> >> 'Documentary',4,
> >> 'Family',3,
> >> 'Horror',3,
> >> 'Musical',2,
> >> 'Romance',2,
> >> 'Concert',1,
> >> 'War',1]},
> >> 'facet_dates'=>{}}
> >>
> >> MAP:
> >>
> >> 'facet_counts'=>{
> >> 'facet_queries'=>{},
> >> 'facet_fields'=>{
> >> 'movies_movie_genre_facet'=>{
> >> 'Drama'=>22,
> >> 'Action/Adventure'=>11,
> >> 'Comedy'=>11,
> >> 'Suspense/Thriller'=>11,
> >> 'SciFi/Fantasy'=>5,
> >> 'Animation'=>4,
> >> 'Documentary'=>4,
> >> 'Family'=>3,
> >> 'Horror'=>3,
> >> 'Musical'=>2,
> >> 'Romance'=>2,
> >> 'Concert'=>1,
> >> 'War'=>1}},
> >> 'facet_dates'=>{}}
> >>
> >> Doug
> >>
> >>
> >>
> >> Yonik Seeley wrote:
> >> > On Wed, Mar 5, 2008 at 11:25 AM, Doug Steigerwald
> >> > <ds...@mcclatchyinteractive.com> wrote:
> >> >> If you don't add the json.nl=map to your params, then you can't eval() what you get back in Ruby
> >> >> ("can't convert String into Integer").
> >> >
> >> > Can you show what the problematic ruby output is?
> >> >
> >> > json.nl=map isn't the default because some things need to be ordered,
> >> > and eval of a map in python & ruby looses that order.
> >> >
> >> > -Yonik
> >>
>
Re: JSONRequestWriter
Posted by Doug Steigerwald <ds...@mcclatchyinteractive.com>.
Looks like it's only happening when we use the LocalSolrQueryComponent from localsolr.
rsp.add("response", sdoclist);
sdoclist is a SolrDocumentList. Could that be causing an issue instead of it being just a DocList?
Doug
Yonik Seeley wrote:
> The output you showed is indeed incorrect, but I can't reproduce that
> with stock solr.
> Here is a example of what I get:
>
> {
> 'responseHeader'=>{
> 'status'=>0,
> 'QTime'=>16,
> 'params'=>{
> 'wt'=>'ruby',
> 'indent'=>'true',
> 'q'=>'*:*',
> 'facet'=>'true',
> 'highlight'=>'true'}},
> 'response'=>{'numFound'=>0,'start'=>0,'docs'=>[]
> },
> 'facet_counts'=>{
> 'facet_queries'=>{},
> 'facet_fields'=>{},
> 'facet_dates'=>{}}}
>
>
> -Yonik
>
> On Wed, Mar 5, 2008 at 12:00 PM, Doug Steigerwald
> <ds...@mcclatchyinteractive.com> wrote:
>> Sure.
>>
>> The default (json.nl=flat):
>>
>> 'response',{'numFound'=>41,'start'=>0,............
>>
>> Adding json.nl=map makes output correct:
>>
>> 'response'=>{'numFound'=>41,'start'=>0,............
>>
>> This also changes facet output (which was evaluating fine):
>>
>> FLAT:
>>
>> 'facet_counts',{
>> 'facet_queries'=>{},
>> 'facet_fields'=>{
>> 'movies_movie_genre_facet'=>[
>> 'Drama',22,
>> 'Action/Adventure',11,
>> 'Comedy',11,
>> 'Suspense/Thriller',11,
>> 'SciFi/Fantasy',5,
>> 'Animation',4,
>> 'Documentary',4,
>> 'Family',3,
>> 'Horror',3,
>> 'Musical',2,
>> 'Romance',2,
>> 'Concert',1,
>> 'War',1]},
>> 'facet_dates'=>{}}
>>
>> MAP:
>>
>> 'facet_counts'=>{
>> 'facet_queries'=>{},
>> 'facet_fields'=>{
>> 'movies_movie_genre_facet'=>{
>> 'Drama'=>22,
>> 'Action/Adventure'=>11,
>> 'Comedy'=>11,
>> 'Suspense/Thriller'=>11,
>> 'SciFi/Fantasy'=>5,
>> 'Animation'=>4,
>> 'Documentary'=>4,
>> 'Family'=>3,
>> 'Horror'=>3,
>> 'Musical'=>2,
>> 'Romance'=>2,
>> 'Concert'=>1,
>> 'War'=>1}},
>> 'facet_dates'=>{}}
>>
>> Doug
>>
>>
>>
>> Yonik Seeley wrote:
>> > On Wed, Mar 5, 2008 at 11:25 AM, Doug Steigerwald
>> > <ds...@mcclatchyinteractive.com> wrote:
>> >> If you don't add the json.nl=map to your params, then you can't eval() what you get back in Ruby
>> >> ("can't convert String into Integer").
>> >
>> > Can you show what the problematic ruby output is?
>> >
>> > json.nl=map isn't the default because some things need to be ordered,
>> > and eval of a map in python & ruby looses that order.
>> >
>> > -Yonik
>>
Re: JSONRequestWriter
Posted by Yonik Seeley <yo...@apache.org>.
The output you showed is indeed incorrect, but I can't reproduce that
with stock solr.
Here is a example of what I get:
{
'responseHeader'=>{
'status'=>0,
'QTime'=>16,
'params'=>{
'wt'=>'ruby',
'indent'=>'true',
'q'=>'*:*',
'facet'=>'true',
'highlight'=>'true'}},
'response'=>{'numFound'=>0,'start'=>0,'docs'=>[]
},
'facet_counts'=>{
'facet_queries'=>{},
'facet_fields'=>{},
'facet_dates'=>{}}}
-Yonik
On Wed, Mar 5, 2008 at 12:00 PM, Doug Steigerwald
<ds...@mcclatchyinteractive.com> wrote:
> Sure.
>
> The default (json.nl=flat):
>
> 'response',{'numFound'=>41,'start'=>0,............
>
> Adding json.nl=map makes output correct:
>
> 'response'=>{'numFound'=>41,'start'=>0,............
>
> This also changes facet output (which was evaluating fine):
>
> FLAT:
>
> 'facet_counts',{
> 'facet_queries'=>{},
> 'facet_fields'=>{
> 'movies_movie_genre_facet'=>[
> 'Drama',22,
> 'Action/Adventure',11,
> 'Comedy',11,
> 'Suspense/Thriller',11,
> 'SciFi/Fantasy',5,
> 'Animation',4,
> 'Documentary',4,
> 'Family',3,
> 'Horror',3,
> 'Musical',2,
> 'Romance',2,
> 'Concert',1,
> 'War',1]},
> 'facet_dates'=>{}}
>
> MAP:
>
> 'facet_counts'=>{
> 'facet_queries'=>{},
> 'facet_fields'=>{
> 'movies_movie_genre_facet'=>{
> 'Drama'=>22,
> 'Action/Adventure'=>11,
> 'Comedy'=>11,
> 'Suspense/Thriller'=>11,
> 'SciFi/Fantasy'=>5,
> 'Animation'=>4,
> 'Documentary'=>4,
> 'Family'=>3,
> 'Horror'=>3,
> 'Musical'=>2,
> 'Romance'=>2,
> 'Concert'=>1,
> 'War'=>1}},
> 'facet_dates'=>{}}
>
> Doug
>
>
>
> Yonik Seeley wrote:
> > On Wed, Mar 5, 2008 at 11:25 AM, Doug Steigerwald
> > <ds...@mcclatchyinteractive.com> wrote:
> >> If you don't add the json.nl=map to your params, then you can't eval() what you get back in Ruby
> >> ("can't convert String into Integer").
> >
> > Can you show what the problematic ruby output is?
> >
> > json.nl=map isn't the default because some things need to be ordered,
> > and eval of a map in python & ruby looses that order.
> >
> > -Yonik
>
Re: JSONRequestWriter
Posted by Doug Steigerwald <ds...@mcclatchyinteractive.com>.
Sure.
The default (json.nl=flat):
'response',{'numFound'=>41,'start'=>0,............
Adding json.nl=map makes output correct:
'response'=>{'numFound'=>41,'start'=>0,............
This also changes facet output (which was evaluating fine):
FLAT:
'facet_counts',{
'facet_queries'=>{},
'facet_fields'=>{
'movies_movie_genre_facet'=>[
'Drama',22,
'Action/Adventure',11,
'Comedy',11,
'Suspense/Thriller',11,
'SciFi/Fantasy',5,
'Animation',4,
'Documentary',4,
'Family',3,
'Horror',3,
'Musical',2,
'Romance',2,
'Concert',1,
'War',1]},
'facet_dates'=>{}}
MAP:
'facet_counts'=>{
'facet_queries'=>{},
'facet_fields'=>{
'movies_movie_genre_facet'=>{
'Drama'=>22,
'Action/Adventure'=>11,
'Comedy'=>11,
'Suspense/Thriller'=>11,
'SciFi/Fantasy'=>5,
'Animation'=>4,
'Documentary'=>4,
'Family'=>3,
'Horror'=>3,
'Musical'=>2,
'Romance'=>2,
'Concert'=>1,
'War'=>1}},
'facet_dates'=>{}}
Doug
Yonik Seeley wrote:
> On Wed, Mar 5, 2008 at 11:25 AM, Doug Steigerwald
> <ds...@mcclatchyinteractive.com> wrote:
>> If you don't add the json.nl=map to your params, then you can't eval() what you get back in Ruby
>> ("can't convert String into Integer").
>
> Can you show what the problematic ruby output is?
>
> json.nl=map isn't the default because some things need to be ordered,
> and eval of a map in python & ruby looses that order.
>
> -Yonik
Re: JSONRequestWriter
Posted by Yonik Seeley <yo...@apache.org>.
On Wed, Mar 5, 2008 at 11:25 AM, Doug Steigerwald
<ds...@mcclatchyinteractive.com> wrote:
> If you don't add the json.nl=map to your params, then you can't eval() what you get back in Ruby
> ("can't convert String into Integer").
Can you show what the problematic ruby output is?
json.nl=map isn't the default because some things need to be ordered,
and eval of a map in python & ruby looses that order.
-Yonik
Re: JSONRequestWriter
Posted by Doug Steigerwald <ds...@mcclatchyinteractive.com>.
Note that we now have to add a default param to the requestHandler:
<requestHandler name="/search" class="org.apache.solr.handler.component.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<str name="json.nl">map</str>
</lst>
<arr name="components">
<str>collapse</str>
<str>localsolr</str>
<str>facet</str>
</arr>
</requestHandler>
If you don't add the json.nl=map to your params, then you can't eval() what you get back in Ruby
("can't convert String into Integer"). Not sure if this can be put into the RubyResponseWriter as a
default. Also not sure if this an issue with the python writer either (since I don't use python).
Doug
Yonik Seeley wrote:
> Thanks Doug, I just checked in your fix.
> This was a recent bug... writing of SolrDocument was recently added
> and is not touched by normal code paths, except for distributed
> search.
>
> -Yonik
>
> On Wed, Mar 5, 2008 at 9:29 AM, Doug Steigerwald
> <ds...@mcclatchyinteractive.com> wrote:
>> We're using localsolr and the RubyResponseWriter. When we do a request with the localsolr component
>> in our requestHandler we're seeing issues with the display of a multivalued field when it only has
>> one value.
>>
>> 'class'=>['showtime']'showtime', <------
>> 'genre'=>['Drama',
>> 'Suspsense/Triller'],
>>
>> With no localsolr component it works fine.
>>
>> Looks like the issue is with the JSONRequestWriter.writeSolrDocument(). Here's the small patch for
>> it that seems to fix it.
>>
>> Index: src/java/org/apache/solr/request/JSONResponseWriter.java
>> ===================================================================
>> --- src/java/org/apache/solr/request/JSONResponseWriter.java (revision 614955)
>> +++ src/java/org/apache/solr/request/JSONResponseWriter.java (working copy)
>> @@ -416,7 +416,7 @@
>> writeVal(fname, val);
>> writeArrayCloser();
>> }
>> - writeVal(fname, val);
>> + else writeVal(fname, val);
>> }
>>
>> if (pseudoFields !=null && pseudoFields.size()>0) {
>>
>>
>> We're running solr trunk r614955 (Jan 23rd), and r75 of localsolr.
>>
>> Result snippet with the patch:
>>
>> 'class'=>['showtime'],
>> 'genre'=>['Drama',
>> 'Suspsense/Triller'],
>>
>> Has anyone come across an issue like this? Is this fixed in a newer build of Solr? It looks like
>> we'd still need this patch even in a build of the solr trunk from yesterday, but maybe not.
>>
>> --
>> Doug Steigerwald
>> Software Developer
>> McClatchy Interactive
>> dsteigerwald@mcclatchyinteractive.com
>> 919.861.1287
>>
Re: JSONRequestWriter
Posted by Yonik Seeley <yo...@apache.org>.
Thanks Doug, I just checked in your fix.
This was a recent bug... writing of SolrDocument was recently added
and is not touched by normal code paths, except for distributed
search.
-Yonik
On Wed, Mar 5, 2008 at 9:29 AM, Doug Steigerwald
<ds...@mcclatchyinteractive.com> wrote:
> We're using localsolr and the RubyResponseWriter. When we do a request with the localsolr component
> in our requestHandler we're seeing issues with the display of a multivalued field when it only has
> one value.
>
> 'class'=>['showtime']'showtime', <------
> 'genre'=>['Drama',
> 'Suspsense/Triller'],
>
> With no localsolr component it works fine.
>
> Looks like the issue is with the JSONRequestWriter.writeSolrDocument(). Here's the small patch for
> it that seems to fix it.
>
> Index: src/java/org/apache/solr/request/JSONResponseWriter.java
> ===================================================================
> --- src/java/org/apache/solr/request/JSONResponseWriter.java (revision 614955)
> +++ src/java/org/apache/solr/request/JSONResponseWriter.java (working copy)
> @@ -416,7 +416,7 @@
> writeVal(fname, val);
> writeArrayCloser();
> }
> - writeVal(fname, val);
> + else writeVal(fname, val);
> }
>
> if (pseudoFields !=null && pseudoFields.size()>0) {
>
>
> We're running solr trunk r614955 (Jan 23rd), and r75 of localsolr.
>
> Result snippet with the patch:
>
> 'class'=>['showtime'],
> 'genre'=>['Drama',
> 'Suspsense/Triller'],
>
> Has anyone come across an issue like this? Is this fixed in a newer build of Solr? It looks like
> we'd still need this patch even in a build of the solr trunk from yesterday, but maybe not.
>
> --
> Doug Steigerwald
> Software Developer
> McClatchy Interactive
> dsteigerwald@mcclatchyinteractive.com
> 919.861.1287
>