Result docs missing only when shards parameter present in query?

[Bill Havanki <ha...@gmail.com>]

I had the same problem as the original poster did two years ago (!), but
with Solr 3.4.0:

> I cannot get hits back and do not get a correct total number of records
when using shard searching.

When performing a sharded query, I would get empty / missing results - no
documents at all. Querying each shard individually worked, but anything
with the "shards" parameter yielded no result documents.

I was able to get results back by updating my schema to include
multiValued="false" for the unique key field.

The problem I was seeing was that, when Solr was formulating the queries to
go get records from each shard, it was including square brackets around the
ids it was asking for, e.g.:

...q=123&ids=[ID1],[ID2],[ID3]&...

I delved into the Solr code and saw that this query string was being formed
(in QueryComponent.createRetrieveDocs()) by simply calling toString() on
the unique key field value for each document it wanted to get. My guess is
that the value objects somehow were ArrayLists (or something like that) and
not Strings, so those annoying square brackets showed up via toString(). By
emphasizing in the schema that the field was single-valued, those lists
would hopefully stop appearing, and I think they did. At least the brackets
went away.

Here's the relevant QueryComponent code (again, 3.4.0 - it's the same in
3.6.0, didn't check 4):

ArrayList<String> ids = new ArrayList<String>(shardDocs.size());
for (ShardDoc shardDoc : shardDocs) {
// TODO: depending on the type, we may need more tha a simple toString()?
  ids.add(shardDoc.id.toString());
}
sreq.params.add(ShardParams.IDS, StrUtils.join(ids, ','));

The comment in there seems to fit my theory. :)

Bill

Re: Result docs missing only when shards parameter present in query?

[Erick Erickson <er...@gmail.com>]

A multiValued <uniqueKey> really doesn't make any sense. But your
log file should have something in it like this:
SEVERE: uniqueKey should not be multivalued
although it _is_ a bit hard to see on startup unless you've suppressed
the INFO level output.

See: https://issues.apache.org/jira/browse/SOLR-1570

Best
Erick

On Tue, Jul 17, 2012 at 9:24 AM, Bill Havanki <ha...@gmail.com> wrote:
> I had the same problem as the original poster did two years ago (!), but
> with Solr 3.4.0:
>
>> I cannot get hits back and do not get a correct total number of records
> when using shard searching.
>
> When performing a sharded query, I would get empty / missing results - no
> documents at all. Querying each shard individually worked, but anything
> with the "shards" parameter yielded no result documents.
>
> I was able to get results back by updating my schema to include
> multiValued="false" for the unique key field.
>
> The problem I was seeing was that, when Solr was formulating the queries to
> go get records from each shard, it was including square brackets around the
> ids it was asking for, e.g.:
>
> ...q=123&ids=[ID1],[ID2],[ID3]&...
>
> I delved into the Solr code and saw that this query string was being formed
> (in QueryComponent.createRetrieveDocs()) by simply calling toString() on
> the unique key field value for each document it wanted to get. My guess is
> that the value objects somehow were ArrayLists (or something like that) and
> not Strings, so those annoying square brackets showed up via toString(). By
> emphasizing in the schema that the field was single-valued, those lists
> would hopefully stop appearing, and I think they did. At least the brackets
> went away.
>
> Here's the relevant QueryComponent code (again, 3.4.0 - it's the same in
> 3.6.0, didn't check 4):
>
> ArrayList<String> ids = new ArrayList<String>(shardDocs.size());
> for (ShardDoc shardDoc : shardDocs) {
> // TODO: depending on the type, we may need more tha a simple toString()?
>   ids.add(shardDoc.id.toString());
> }
> sreq.params.add(ShardParams.IDS, StrUtils.join(ids, ','));
>
> The comment in there seems to fit my theory. :)
>
> Bill