You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Paul Rosen <pa...@performantsoftware.com> on 2009/09/10 23:05:32 UTC
shards and facet_count
Hi again,
I've mostly gotten the multicore working except for one detail.
(I'm using solr 1.3 and solr-ruby 0.0.6 in a rails project.)
I've done a few queries and I appear to be able to get hits from either
core. (yeah!)
I'm forming my request like this:
req = Solr::Request::Standard.new(
:start => start,
:rows => max,
:sort => sort_param,
:query => query,
:filter_queries => filter_queries,
:field_list => @field_list,
:facets => {:fields => @facet_fields, :mincount => 1, :missing =>
true, :limit => -1},
:highlighting => {:field_list => ['text'], :fragment_size => 600},
:shards => @cores)
If I leave ":shards => @cores" out, then the response includes:
'facet_counts' => {
'facet_dates' => {},
'facet_queries' => {},
'facet_fields' => { 'myfacet' => [ etc...], etc... }
which is what I expect.
If I add the ":shards => @cores" back in (so that I'm doing the exact
request above), I get:
'facet_counts' => {
'facet_dates' => {},
'facet_queries' => {},
'facet_fields' => {}
so I've lost my facet information.
Why would it correctly find my documents, but not report the facet info?
Thanks,
Paul
Re: shards and facet_count
Posted by Paul Rosen <pa...@performantsoftware.com>.
Sorry for the long delay in responding, but I've just gotten back to
this problem...
I got the solr 1.4 nightly and the problem went away, so I guess it is a
solr 1.3 bug.
Thanks for all the input!
Lance Norskog wrote:
> Paul, can you create an HTTP url that does this exact query? With
> multiple shards and facet requests? And that does what you expect?
> That would help the Ruby Dudes to figure out the discrepancy.
>
> Lance
>
> On Fri, Sep 18, 2009 at 7:01 PM, Yonik Seeley
> <yo...@lucidimagination.com> wrote:
>> On Fri, Sep 18, 2009 at 5:58 AM, Erik Hatcher <er...@gmail.com> wrote:
>>> It is strange that you get facet=false calls in there, but maybe this is
>>> just normal distributed search protocol in one of the phases?
>> Right, on the second phase of a distrib request, additional faceting
>> may not be needed.
>>
>> But it looks like the distributed request is being directed at two
>> different handlers rather than two different servers or cores?
>> shards=localhost:8983/solr/resources,localhost:8983/solr/exhibits
>>
>> I've never tried this, but from the log file, it doesn't look like the
>> sub-requests are going to those different handlers since the path is
>> always path=/select
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>
>
>
Re: shards and facet_count
Posted by Lance Norskog <go...@gmail.com>.
Paul, can you create an HTTP url that does this exact query? With
multiple shards and facet requests? And that does what you expect?
That would help the Ruby Dudes to figure out the discrepancy.
Lance
On Fri, Sep 18, 2009 at 7:01 PM, Yonik Seeley
<yo...@lucidimagination.com> wrote:
> On Fri, Sep 18, 2009 at 5:58 AM, Erik Hatcher <er...@gmail.com> wrote:
>> It is strange that you get facet=false calls in there, but maybe this is
>> just normal distributed search protocol in one of the phases?
>
> Right, on the second phase of a distrib request, additional faceting
> may not be needed.
>
> But it looks like the distributed request is being directed at two
> different handlers rather than two different servers or cores?
> shards=localhost:8983/solr/resources,localhost:8983/solr/exhibits
>
> I've never tried this, but from the log file, it doesn't look like the
> sub-requests are going to those different handlers since the path is
> always path=/select
>
> -Yonik
> http://www.lucidimagination.com
>
--
Lance Norskog
goksron@gmail.com
Re: shards and facet_count
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Fri, Sep 18, 2009 at 5:58 AM, Erik Hatcher <er...@gmail.com> wrote:
> It is strange that you get facet=false calls in there, but maybe this is
> just normal distributed search protocol in one of the phases?
Right, on the second phase of a distrib request, additional faceting
may not be needed.
But it looks like the distributed request is being directed at two
different handlers rather than two different servers or cores?
shards=localhost:8983/solr/resources,localhost:8983/solr/exhibits
I've never tried this, but from the log file, it doesn't look like the
sub-requests are going to those different handlers since the path is
always path=/select
-Yonik
http://www.lucidimagination.com
Re: shards and facet_count
Posted by Erik Hatcher <er...@gmail.com>.
On Sep 17, 2009, at 6:14 PM, Lance Norskog wrote:
> Yes. &facet=false means "don't do any faceting". This is why you don't
> get any facet data back. This is probably a bug in the solr-ruby code.
> Version number 0.0.x is probably a hint about its production-ready
> status :)
Actually solr-ruby is plenty suitable for production use - it's pretty
straightforward mapping from Rubyish stuff to Solr requests. Not
really much magic in there. It's 0.0.x version number is really my
own laziness (or swampedness) in getting it polished in a form I'd
like (I'm a perfectionist with no time on his hands, a frustrating
existence). The RSolr API has some features I'd like to pull over
into solr-ruby and do some refactoring one of these days in my copious
free time, but in general solr-ruby works fine.
It is strange that you get facet=false calls in there, but maybe this
is just normal distributed search protocol in one of the phases?
Erik
> On Mon, Sep 14, 2009 at 6:46 AM, Paul Rosen <paul@performantsoftware.com
> > wrote:
>> Shalin Shekhar Mangar wrote:
>>>
>>> On Fri, Sep 11, 2009 at 2:35 AM, Paul Rosen
>>> <pa...@performantsoftware.com>wrote:
>>>
>>>> Hi again,
>>>>
>>>> I've mostly gotten the multicore working except for one detail.
>>>>
>>>> (I'm using solr 1.3 and solr-ruby 0.0.6 in a rails project.)
>>>>
>>>> I've done a few queries and I appear to be able to get hits from
>>>> either
>>>> core. (yeah!)
>>>>
>>>> I'm forming my request like this:
>>>>
>>>> req = Solr::Request::Standard.new(
>>>> :start => start,
>>>> :rows => max,
>>>> :sort => sort_param,
>>>> :query => query,
>>>> :filter_queries => filter_queries,
>>>> :field_list => @field_list,
>>>> :facets => {:fields => @facet_fields, :mincount => 1, :missing
>>>> => true,
>>>> :limit => -1},
>>>> :highlighting => {:field_list => ['text'], :fragment_size => 600},
>>>> :shards => @cores)
>>>>
>>>> If I leave ":shards => @cores" out, then the response includes:
>>>>
>>>> 'facet_counts' => {
>>>> 'facet_dates' => {},
>>>> 'facet_queries' => {},
>>>> 'facet_fields' => { 'myfacet' => [ etc...], etc... }
>>>>
>>>> which is what I expect.
>>>>
>>>> If I add the ":shards => @cores" back in (so that I'm doing the
>>>> exact
>>>> request above), I get:
>>>>
>>>> 'facet_counts' => {
>>>> 'facet_dates' => {},
>>>> 'facet_queries' => {},
>>>> 'facet_fields' => {}
>>>>
>>>> so I've lost my facet information.
>>>>
>>>> Why would it correctly find my documents, but not report the
>>>> facet info?
>>>>
>>>
>>> I'm not a ruby guy but the response format in both the cases is
>>> exactly
>>> the
>>> same so I don't think there is any problem with the ruby client
>>> parsing.
>>> Can
>>> you check the Solr logs to see if there were any exceptions when
>>> you sent
>>> the shards parameter?
>>>
>>
>> I don't see any exceptions. The solr activity is pretty different
>> for the
>> two cases. Without the shards, it makes one call that looks
>> something like
>> this (I ellipsed the id and field parameters for clarity):
>>
>> Sep 14, 2009 9:32:09 AM org.apache.solr.core.SolrCore execute
>> INFO: [resources] webapp=/solr path=/select
>> params
>> =
>> {facet
>> .limit
>> =
>> -1
>> &wt
>> =
>> ruby
>> &rows
>> =
>> 30
>> &start
>> =
>> 0
>> &facet
>> =
>> true
>> &facet
>> .mincount
>> =
>> 1
>> &q
>> =
>> (rossetti
>> )&fl
>> =
>> archive
>> ,...,license
>> &qt
>> =
>> standard
>> &facet
>> .missing
>> =
>> true
>> &hl
>> .fl
>> =
>> text
>> &facet
>> .field
>> =
>> genre
>> &facet.field=archive&facet.field=freeculture&hl.fragsize=600&hl=true}
>> hits=27 status=0 QTime=6
>>
>> Note that "facet=true".
>>
>> With the shards, it has five lines for the single call that I make:
>>
>> Sep 14, 2009 9:37:18 AM org.apache.solr.core.SolrCore execute
>> INFO: [exhibits] webapp=/solr path=/select
>> params
>> =
>> {wt
>> =
>> javabin
>> &rows
>> =
>> 30
>> &start
>> =
>> 0
>> &facet
>> =
>> true
>> &fl
>> =
>> uri
>> ,score
>> &q
>> =
>> (rossetti
>> )&version
>> =
>> 2.2
>> &isShard
>> =
>> true
>> &facet
>> .missing
>> =
>> true
>> &hl
>> .fl
>> =
>> text
>> &fsv
>> =
>> true
>> &hl
>> .fragsize
>> =
>> 600
>> &facet
>> .field=genre&facet.field=archive&facet.field=freeculture&hl=false}
>> hits=6 status=0 QTime=0
>>
>> Sep 14, 2009 9:37:18 AM org.apache.solr.core.SolrCore execute
>> INFO: [resources] webapp=/solr path=/select
>> params
>> =
>> {wt
>> =
>> javabin
>> &rows
>> =
>> 30
>> &start
>> =
>> 0
>> &facet
>> =
>> true
>> &fl
>> =
>> uri
>> ,score
>> &q
>> =
>> (rossetti
>> )&version
>> =
>> 2.2
>> &isShard
>> =
>> true
>> &facet
>> .missing
>> =
>> true
>> &hl
>> .fl
>> =
>> text
>> &fsv
>> =
>> true
>> &hl
>> .fragsize
>> =
>> 600
>> &facet
>> .field=genre&facet.field=archive&facet.field=freeculture&hl=false}
>> hits=27 status=0 QTime=3
>>
>> Sep 14, 2009 9:37:18 AM org.apache.solr.core.SolrCore execute
>> INFO: [resources] webapp=/solr path=/select
>> params={facet.limit=-1&wt=javabin&rows=30&start=0&ids=...,...&facet=false&facet.mincount=1&q=(rossetti)&fl=archive,...,uri&version=2.2&facet.missing=true&isShard=true&hl.fl=text&facet.field=genre&facet.field=archive&facet.field=freeculture&hl.fragsize=600&hl=true}
>> status=0 QTime=35
>>
>> Sep 14, 2009 9:37:18 AM org.apache.solr.core.SolrCore execute
>> INFO: [exhibits] webapp=/solr path=/select
>> params={facet.limit=-1&wt=javabin&rows=30&start=0&ids=...,...&facet=false&facet.mincount=1&q=(rossetti)&fl=archive,...,uri&version=2.2&facet.missing=true&isShard=true&hl.fl=text&facet.field=genre&facet.field=archive&facet.field=freeculture&hl.fragsize=600&hl=true}
>> status=0 QTime=41
>>
>> Sep 14, 2009 9:37:18 AM org.apache.solr.core.SolrCore execute
>> INFO: [resources] webapp=/solr path=/select
>> params={facet.limit=-1&wt=ruby&rows=30&start=0&facet=true&facet.mincount=1&q=(rossetti)&fl=archive,...,license&qt=standard&facet.missing=true&hl.fl=text&facet.field=genre&facet.field=archive&facet.field=freeculture&hl.fragsize=600&hl=true&shards=localhost:8983/solr/resources,localhost:8983/solr/exhibits}
>> status=0 QTime=57
>>
>> Note that on the third and fourth lines, "facet=false". Is that
>> significant?
>>
>> Thanks,
>> Paul
>>
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
Re: shards and facet_count
Posted by Lance Norskog <go...@gmail.com>.
Yes. &facet=false means "don't do any faceting". This is why you don't
get any facet data back. This is probably a bug in the solr-ruby code.
Version number 0.0.x is probably a hint about its production-ready
status :)
A distributed search system is easier to administer if you add a new
core that is the "distributor". It has no Solr data, but the
solrconfig.xml includes search requestHandlers that always search a
set of other cores. This is done with an "<invariants>" element that
hard-codes "cores" as your list.
On Mon, Sep 14, 2009 at 6:46 AM, Paul Rosen <pa...@performantsoftware.com> wrote:
> Shalin Shekhar Mangar wrote:
>>
>> On Fri, Sep 11, 2009 at 2:35 AM, Paul Rosen
>> <pa...@performantsoftware.com>wrote:
>>
>>> Hi again,
>>>
>>> I've mostly gotten the multicore working except for one detail.
>>>
>>> (I'm using solr 1.3 and solr-ruby 0.0.6 in a rails project.)
>>>
>>> I've done a few queries and I appear to be able to get hits from either
>>> core. (yeah!)
>>>
>>> I'm forming my request like this:
>>>
>>> req = Solr::Request::Standard.new(
>>> :start => start,
>>> :rows => max,
>>> :sort => sort_param,
>>> :query => query,
>>> :filter_queries => filter_queries,
>>> :field_list => @field_list,
>>> :facets => {:fields => @facet_fields, :mincount => 1, :missing => true,
>>> :limit => -1},
>>> :highlighting => {:field_list => ['text'], :fragment_size => 600},
>>> :shards => @cores)
>>>
>>> If I leave ":shards => @cores" out, then the response includes:
>>>
>>> 'facet_counts' => {
>>> 'facet_dates' => {},
>>> 'facet_queries' => {},
>>> 'facet_fields' => { 'myfacet' => [ etc...], etc... }
>>>
>>> which is what I expect.
>>>
>>> If I add the ":shards => @cores" back in (so that I'm doing the exact
>>> request above), I get:
>>>
>>> 'facet_counts' => {
>>> 'facet_dates' => {},
>>> 'facet_queries' => {},
>>> 'facet_fields' => {}
>>>
>>> so I've lost my facet information.
>>>
>>> Why would it correctly find my documents, but not report the facet info?
>>>
>>
>> I'm not a ruby guy but the response format in both the cases is exactly
>> the
>> same so I don't think there is any problem with the ruby client parsing.
>> Can
>> you check the Solr logs to see if there were any exceptions when you sent
>> the shards parameter?
>>
>
> I don't see any exceptions. The solr activity is pretty different for the
> two cases. Without the shards, it makes one call that looks something like
> this (I ellipsed the id and field parameters for clarity):
>
> Sep 14, 2009 9:32:09 AM org.apache.solr.core.SolrCore execute
> INFO: [resources] webapp=/solr path=/select
> params={facet.limit=-1&wt=ruby&rows=30&start=0&facet=true&facet.mincount=1&q=(rossetti)&fl=archive,...,license&qt=standard&facet.missing=true&hl.fl=text&facet.field=genre&facet.field=archive&facet.field=freeculture&hl.fragsize=600&hl=true}
> hits=27 status=0 QTime=6
>
> Note that "facet=true".
>
> With the shards, it has five lines for the single call that I make:
>
> Sep 14, 2009 9:37:18 AM org.apache.solr.core.SolrCore execute
> INFO: [exhibits] webapp=/solr path=/select
> params={wt=javabin&rows=30&start=0&facet=true&fl=uri,score&q=(rossetti)&version=2.2&isShard=true&facet.missing=true&hl.fl=text&fsv=true&hl.fragsize=600&facet.field=genre&facet.field=archive&facet.field=freeculture&hl=false}
> hits=6 status=0 QTime=0
>
> Sep 14, 2009 9:37:18 AM org.apache.solr.core.SolrCore execute
> INFO: [resources] webapp=/solr path=/select
> params={wt=javabin&rows=30&start=0&facet=true&fl=uri,score&q=(rossetti)&version=2.2&isShard=true&facet.missing=true&hl.fl=text&fsv=true&hl.fragsize=600&facet.field=genre&facet.field=archive&facet.field=freeculture&hl=false}
> hits=27 status=0 QTime=3
>
> Sep 14, 2009 9:37:18 AM org.apache.solr.core.SolrCore execute
> INFO: [resources] webapp=/solr path=/select
> params={facet.limit=-1&wt=javabin&rows=30&start=0&ids=...,...&facet=false&facet.mincount=1&q=(rossetti)&fl=archive,...,uri&version=2.2&facet.missing=true&isShard=true&hl.fl=text&facet.field=genre&facet.field=archive&facet.field=freeculture&hl.fragsize=600&hl=true}
> status=0 QTime=35
>
> Sep 14, 2009 9:37:18 AM org.apache.solr.core.SolrCore execute
> INFO: [exhibits] webapp=/solr path=/select
> params={facet.limit=-1&wt=javabin&rows=30&start=0&ids=...,...&facet=false&facet.mincount=1&q=(rossetti)&fl=archive,...,uri&version=2.2&facet.missing=true&isShard=true&hl.fl=text&facet.field=genre&facet.field=archive&facet.field=freeculture&hl.fragsize=600&hl=true}
> status=0 QTime=41
>
> Sep 14, 2009 9:37:18 AM org.apache.solr.core.SolrCore execute
> INFO: [resources] webapp=/solr path=/select
> params={facet.limit=-1&wt=ruby&rows=30&start=0&facet=true&facet.mincount=1&q=(rossetti)&fl=archive,...,license&qt=standard&facet.missing=true&hl.fl=text&facet.field=genre&facet.field=archive&facet.field=freeculture&hl.fragsize=600&hl=true&shards=localhost:8983/solr/resources,localhost:8983/solr/exhibits}
> status=0 QTime=57
>
> Note that on the third and fourth lines, "facet=false". Is that significant?
>
> Thanks,
> Paul
>
>
--
Lance Norskog
goksron@gmail.com
Re: shards and facet_count
Posted by Paul Rosen <pa...@performantsoftware.com>.
Shalin Shekhar Mangar wrote:
> On Fri, Sep 11, 2009 at 2:35 AM, Paul Rosen <pa...@performantsoftware.com>wrote:
>
>> Hi again,
>>
>> I've mostly gotten the multicore working except for one detail.
>>
>> (I'm using solr 1.3 and solr-ruby 0.0.6 in a rails project.)
>>
>> I've done a few queries and I appear to be able to get hits from either
>> core. (yeah!)
>>
>> I'm forming my request like this:
>>
>> req = Solr::Request::Standard.new(
>> :start => start,
>> :rows => max,
>> :sort => sort_param,
>> :query => query,
>> :filter_queries => filter_queries,
>> :field_list => @field_list,
>> :facets => {:fields => @facet_fields, :mincount => 1, :missing => true,
>> :limit => -1},
>> :highlighting => {:field_list => ['text'], :fragment_size => 600},
>> :shards => @cores)
>>
>> If I leave ":shards => @cores" out, then the response includes:
>>
>> 'facet_counts' => {
>> 'facet_dates' => {},
>> 'facet_queries' => {},
>> 'facet_fields' => { 'myfacet' => [ etc...], etc... }
>>
>> which is what I expect.
>>
>> If I add the ":shards => @cores" back in (so that I'm doing the exact
>> request above), I get:
>>
>> 'facet_counts' => {
>> 'facet_dates' => {},
>> 'facet_queries' => {},
>> 'facet_fields' => {}
>>
>> so I've lost my facet information.
>>
>> Why would it correctly find my documents, but not report the facet info?
>>
>
> I'm not a ruby guy but the response format in both the cases is exactly the
> same so I don't think there is any problem with the ruby client parsing. Can
> you check the Solr logs to see if there were any exceptions when you sent
> the shards parameter?
>
I don't see any exceptions. The solr activity is pretty different for
the two cases. Without the shards, it makes one call that looks
something like this (I ellipsed the id and field parameters for clarity):
Sep 14, 2009 9:32:09 AM org.apache.solr.core.SolrCore execute
INFO: [resources] webapp=/solr path=/select
params={facet.limit=-1&wt=ruby&rows=30&start=0&facet=true&facet.mincount=1&q=(rossetti)&fl=archive,...,license&qt=standard&facet.missing=true&hl.fl=text&facet.field=genre&facet.field=archive&facet.field=freeculture&hl.fragsize=600&hl=true}
hits=27 status=0 QTime=6
Note that "facet=true".
With the shards, it has five lines for the single call that I make:
Sep 14, 2009 9:37:18 AM org.apache.solr.core.SolrCore execute
INFO: [exhibits] webapp=/solr path=/select
params={wt=javabin&rows=30&start=0&facet=true&fl=uri,score&q=(rossetti)&version=2.2&isShard=true&facet.missing=true&hl.fl=text&fsv=true&hl.fragsize=600&facet.field=genre&facet.field=archive&facet.field=freeculture&hl=false}
hits=6 status=0 QTime=0
Sep 14, 2009 9:37:18 AM org.apache.solr.core.SolrCore execute
INFO: [resources] webapp=/solr path=/select
params={wt=javabin&rows=30&start=0&facet=true&fl=uri,score&q=(rossetti)&version=2.2&isShard=true&facet.missing=true&hl.fl=text&fsv=true&hl.fragsize=600&facet.field=genre&facet.field=archive&facet.field=freeculture&hl=false}
hits=27 status=0 QTime=3
Sep 14, 2009 9:37:18 AM org.apache.solr.core.SolrCore execute
INFO: [resources] webapp=/solr path=/select
params={facet.limit=-1&wt=javabin&rows=30&start=0&ids=...,...&facet=false&facet.mincount=1&q=(rossetti)&fl=archive,...,uri&version=2.2&facet.missing=true&isShard=true&hl.fl=text&facet.field=genre&facet.field=archive&facet.field=freeculture&hl.fragsize=600&hl=true}
status=0 QTime=35
Sep 14, 2009 9:37:18 AM org.apache.solr.core.SolrCore execute
INFO: [exhibits] webapp=/solr path=/select
params={facet.limit=-1&wt=javabin&rows=30&start=0&ids=...,...&facet=false&facet.mincount=1&q=(rossetti)&fl=archive,...,uri&version=2.2&facet.missing=true&isShard=true&hl.fl=text&facet.field=genre&facet.field=archive&facet.field=freeculture&hl.fragsize=600&hl=true}
status=0 QTime=41
Sep 14, 2009 9:37:18 AM org.apache.solr.core.SolrCore execute
INFO: [resources] webapp=/solr path=/select
params={facet.limit=-1&wt=ruby&rows=30&start=0&facet=true&facet.mincount=1&q=(rossetti)&fl=archive,...,license&qt=standard&facet.missing=true&hl.fl=text&facet.field=genre&facet.field=archive&facet.field=freeculture&hl.fragsize=600&hl=true&shards=localhost:8983/solr/resources,localhost:8983/solr/exhibits}
status=0 QTime=57
Note that on the third and fourth lines, "facet=false". Is that significant?
Thanks,
Paul
Re: shards and facet_count
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Fri, Sep 11, 2009 at 2:35 AM, Paul Rosen <pa...@performantsoftware.com>wrote:
> Hi again,
>
> I've mostly gotten the multicore working except for one detail.
>
> (I'm using solr 1.3 and solr-ruby 0.0.6 in a rails project.)
>
> I've done a few queries and I appear to be able to get hits from either
> core. (yeah!)
>
> I'm forming my request like this:
>
> req = Solr::Request::Standard.new(
> :start => start,
> :rows => max,
> :sort => sort_param,
> :query => query,
> :filter_queries => filter_queries,
> :field_list => @field_list,
> :facets => {:fields => @facet_fields, :mincount => 1, :missing => true,
> :limit => -1},
> :highlighting => {:field_list => ['text'], :fragment_size => 600},
> :shards => @cores)
>
> If I leave ":shards => @cores" out, then the response includes:
>
> 'facet_counts' => {
> 'facet_dates' => {},
> 'facet_queries' => {},
> 'facet_fields' => { 'myfacet' => [ etc...], etc... }
>
> which is what I expect.
>
> If I add the ":shards => @cores" back in (so that I'm doing the exact
> request above), I get:
>
> 'facet_counts' => {
> 'facet_dates' => {},
> 'facet_queries' => {},
> 'facet_fields' => {}
>
> so I've lost my facet information.
>
> Why would it correctly find my documents, but not report the facet info?
>
I'm not a ruby guy but the response format in both the cases is exactly the
same so I don't think there is any problem with the ruby client parsing. Can
you check the Solr logs to see if there were any exceptions when you sent
the shards parameter?
--
Regards,
Shalin Shekhar Mangar.