In cloud mode, using implicit router. Leader changed, not available to index data, and no error occurred.

[Jerome Yang <je...@pivotal.io>]

Hi all,

The situation is:
Three hosts, host1, host2, host3. Solr version 6.1 in cloud mode. 8 solr
nodes on each host.

Create a collection using implicit router. Execute index and delete index.
The collection works fine.
Then kill 3 nodes, some of shards change leader.
Then index data to new leaders of shards, and commit. But some of shards
still has 0 documents. And no error occurred.
By checking the log on that leader replica, it did receive the update
request and processed. No error found in the log.

After restart all nodes, everything works fine.

This is a serious bug I think.
Can you confirm it's a bug or not?

Regards,
Jerome

Re: In cloud mode, using implicit router. Leader changed, not available to index data, and no error occurred.

[Jerome Yang <je...@pivotal.io>]

That shard did receive update request, because it shows in the log.
And also commit request.
But no documents indexed.

On Tue, Sep 20, 2016 at 2:26 PM, Jerome Yang <je...@pivotal.io> wrote:

> I'm sure I send documents to that shard. And execute commit.
>
> I also use curl to index, but not error occurred and no documents are
> indexed.
>
> On Mon, Sep 19, 2016 at 11:27 PM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> Are all the documents in the collection? By using implicit router, you are
>> assuming control of what shard each document ends up on. So my
>> guess is that you are not routing the docs to each shard.
>>
>> If you want Solr to automatically assign the shard to a doc, you should
>> be using the default compositeId routing scheme.
>>
>> If you index docs and not all of them are somewhere in the collection,
>> that's a problem, assuming you are routing them properly when using
>> the implicit router.
>>
>> Best,
>> Erick
>>
>> On Sun, Sep 18, 2016 at 8:04 PM, Jerome Yang <je...@pivotal.io> wrote:
>> > Hi all,
>> >
>> > The situation is:
>> > Three hosts, host1, host2, host3. Solr version 6.1 in cloud mode. 8 solr
>> > nodes on each host.
>> >
>> > Create a collection using implicit router. Execute index and delete
>> index.
>> > The collection works fine.
>> > Then kill 3 nodes, some of shards change leader.
>> > Then index data to new leaders of shards, and commit. But some of shards
>> > still has 0 documents. And no error occurred.
>> > By checking the log on that leader replica, it did receive the update
>> > request and processed. No error found in the log.
>> >
>> > After restart all nodes, everything works fine.
>> >
>> > This is a serious bug I think.
>> > Can you confirm it's a bug or not?
>> >
>> > Regards,
>> > Jerome
>>
>
>

Re: In cloud mode, using implicit router. Leader changed, not available to index data, and no error occurred.

[Jerome Yang <je...@pivotal.io>]

I'm sure I send documents to that shard. And execute commit.

I also use curl to index, but not error occurred and no documents are
indexed.

On Mon, Sep 19, 2016 at 11:27 PM, Erick Erickson <er...@gmail.com>
wrote:

> Are all the documents in the collection? By using implicit router, you are
> assuming control of what shard each document ends up on. So my
> guess is that you are not routing the docs to each shard.
>
> If you want Solr to automatically assign the shard to a doc, you should
> be using the default compositeId routing scheme.
>
> If you index docs and not all of them are somewhere in the collection,
> that's a problem, assuming you are routing them properly when using
> the implicit router.
>
> Best,
> Erick
>
> On Sun, Sep 18, 2016 at 8:04 PM, Jerome Yang <je...@pivotal.io> wrote:
> > Hi all,
> >
> > The situation is:
> > Three hosts, host1, host2, host3. Solr version 6.1 in cloud mode. 8 solr
> > nodes on each host.
> >
> > Create a collection using implicit router. Execute index and delete
> index.
> > The collection works fine.
> > Then kill 3 nodes, some of shards change leader.
> > Then index data to new leaders of shards, and commit. But some of shards
> > still has 0 documents. And no error occurred.
> > By checking the log on that leader replica, it did receive the update
> > request and processed. No error found in the log.
> >
> > After restart all nodes, everything works fine.
> >
> > This is a serious bug I think.
> > Can you confirm it's a bug or not?
> >
> > Regards,
> > Jerome
>

Re: In cloud mode, using implicit router. Leader changed, not available to index data, and no error occurred.

[Erick Erickson <er...@gmail.com>]

Are all the documents in the collection? By using implicit router, you are
assuming control of what shard each document ends up on. So my
guess is that you are not routing the docs to each shard.

If you want Solr to automatically assign the shard to a doc, you should
be using the default compositeId routing scheme.

If you index docs and not all of them are somewhere in the collection,
that's a problem, assuming you are routing them properly when using
the implicit router.

Best,
Erick

On Sun, Sep 18, 2016 at 8:04 PM, Jerome Yang <je...@pivotal.io> wrote:
> Hi all,
>
> The situation is:
> Three hosts, host1, host2, host3. Solr version 6.1 in cloud mode. 8 solr
> nodes on each host.
>
> Create a collection using implicit router. Execute index and delete index.
> The collection works fine.
> Then kill 3 nodes, some of shards change leader.
> Then index data to new leaders of shards, and commit. But some of shards
> still has 0 documents. And no error occurred.
> By checking the log on that leader replica, it did receive the update
> request and processed. No error found in the log.
>
> After restart all nodes, everything works fine.
>
> This is a serious bug I think.
> Can you confirm it's a bug or not?
>
> Regards,
> Jerome