PostFilter does not seem to work across shards

[Kevin Osborn <ko...@centraldesktop.com>]

I developed a post filter. My documents to be filtered are on two different
shards. So, in a single-shard environment,
DelegatingCollector.doSetNextReader is called twice. And collect is called
the correct number of times. Everything went well and I got my correct
number of results back.

So, I then tried this filter in a two-shard environment. This time things
did not work well. I am still trying to figure out what is going on, but it
seems like just the first shard is being used. I get the same results no
matter what shard or replica I begin my query on. But it seems like the
results are not being merged.

Although I am still trying to figure out if the second shard is even being
queried.

Are there any known issues with DelegatingCollector and shards?

I don't know if this is related, but I did once get the following error
message as well.

java.lang.UnsupportedOperationException: Query {!cache=false cost=100}
does not implement
createWeight

Re: PostFilter does not seem to work across shards

[Erick Erickson <er...@gmail.com>]

Whew! Thanks for bring this to closure!

Best,
Erick

On Mon, Mar 23, 2015 at 9:18 AM, Kevin Osborn
<ko...@centraldesktop.com> wrote:
> I think I found my issue. It has nothing to do with the post filter. In the
> constructor of my post filter, I am doing a TermQuery do get a single user
> document. I then later intersect this user's permissions with the collected
> documents. So, if the user document is in the shard that I am filtering in,
> it works fine. I retrieve the object and do my intersections. But, on the
> other shard, I don't have my user document. So, I have nothing to intersect
> with.
>
> That is a separate issue that I need to figure out.
>
> On Mon, Mar 23, 2015 at 8:09 AM, Kevin Osborn <ko...@centraldesktop.com>
> wrote:
>
>> A little more information here. I have verified that the post filter is
>> giving me only documents that are in the first shard. Running two shards
>> and a single replica in debug mode also shows that the collect method is
>> only called for documents in the first shard. I never see any indication
>> that the filter is called for any documents on the second shard.
>>
>> On Fri, Mar 20, 2015 at 4:12 PM, Kevin Osborn <ko...@centraldesktop.com>
>> wrote:
>>
>>> I developed a post filter. My documents to be filtered are on two
>>> different shards. So, in a single-shard environment,
>>> DelegatingCollector.doSetNextReader is called twice. And collect is called
>>> the correct number of times. Everything went well and I got my correct
>>> number of results back.
>>>
>>> So, I then tried this filter in a two-shard environment. This time things
>>> did not work well. I am still trying to figure out what is going on, but it
>>> seems like just the first shard is being used. I get the same results no
>>> matter what shard or replica I begin my query on. But it seems like the
>>> results are not being merged.
>>>
>>> Although I am still trying to figure out if the second shard is even
>>> being queried.
>>>
>>> Are there any known issues with DelegatingCollector and shards?
>>>
>>> I don't know if this is related, but I did once get the following error
>>> message as well.
>>>
>>> java.lang.UnsupportedOperationException: Query {!cache=false cost=100} does not implement
>>> createWeight
>>>
>>>
>>

Re: PostFilter does not seem to work across shards

[Kevin Osborn <ko...@centraldesktop.com>]

I think I found my issue. It has nothing to do with the post filter. In the
constructor of my post filter, I am doing a TermQuery do get a single user
document. I then later intersect this user's permissions with the collected
documents. So, if the user document is in the shard that I am filtering in,
it works fine. I retrieve the object and do my intersections. But, on the
other shard, I don't have my user document. So, I have nothing to intersect
with.

That is a separate issue that I need to figure out.

On Mon, Mar 23, 2015 at 8:09 AM, Kevin Osborn <ko...@centraldesktop.com>
wrote:

> A little more information here. I have verified that the post filter is
> giving me only documents that are in the first shard. Running two shards
> and a single replica in debug mode also shows that the collect method is
> only called for documents in the first shard. I never see any indication
> that the filter is called for any documents on the second shard.
>
> On Fri, Mar 20, 2015 at 4:12 PM, Kevin Osborn <ko...@centraldesktop.com>
> wrote:
>
>> I developed a post filter. My documents to be filtered are on two
>> different shards. So, in a single-shard environment,
>> DelegatingCollector.doSetNextReader is called twice. And collect is called
>> the correct number of times. Everything went well and I got my correct
>> number of results back.
>>
>> So, I then tried this filter in a two-shard environment. This time things
>> did not work well. I am still trying to figure out what is going on, but it
>> seems like just the first shard is being used. I get the same results no
>> matter what shard or replica I begin my query on. But it seems like the
>> results are not being merged.
>>
>> Although I am still trying to figure out if the second shard is even
>> being queried.
>>
>> Are there any known issues with DelegatingCollector and shards?
>>
>> I don't know if this is related, but I did once get the following error
>> message as well.
>>
>> java.lang.UnsupportedOperationException: Query {!cache=false cost=100} does not implement
>> createWeight
>>
>>
>

Re: PostFilter does not seem to work across shards

[Kevin Osborn <ko...@centraldesktop.com>]

A little more information here. I have verified that the post filter is
giving me only documents that are in the first shard. Running two shards
and a single replica in debug mode also shows that the collect method is
only called for documents in the first shard. I never see any indication
that the filter is called for any documents on the second shard.

On Fri, Mar 20, 2015 at 4:12 PM, Kevin Osborn <ko...@centraldesktop.com>
wrote:

> I developed a post filter. My documents to be filtered are on two
> different shards. So, in a single-shard environment,
> DelegatingCollector.doSetNextReader is called twice. And collect is called
> the correct number of times. Everything went well and I got my correct
> number of results back.
>
> So, I then tried this filter in a two-shard environment. This time things
> did not work well. I am still trying to figure out what is going on, but it
> seems like just the first shard is being used. I get the same results no
> matter what shard or replica I begin my query on. But it seems like the
> results are not being merged.
>
> Although I am still trying to figure out if the second shard is even being
> queried.
>
> Are there any known issues with DelegatingCollector and shards?
>
> I don't know if this is related, but I did once get the following error
> message as well.
>
> java.lang.UnsupportedOperationException: Query {!cache=false cost=100} does not implement
> createWeight
>
>