Empty documents in Solr\lucene 3.6

[Alexey Kozhemiakin <Al...@epam.com>]

Dear Community,

We've faced a strange data corruption issue with one of our clients old solr setup (3.6).

When we do a query (id:X OR id:Y) we get 2 nodes, one contains normal doc data, another is empty (<doc />).
We've looked inside lucene index using Luke - same story, one of documents is empty.
When we click on 1st document - it shows nothing.
http://snag.gy/O5Lgq.jpg


Probably files for stored data were corrupted? But luke index check says OK.
Any clues how to troubleshoot root cause?

Best regards,
Alexey

Re: Empty documents in Solr\lucene 3.6

[Shawn Heisey <so...@elyograg.org>]

On 4/15/2014 10:22 AM, Alexey Kozhemiakin wrote:
> The system was up and running for long time(months) without any updates.
> There was no crashes for sure, at least support team says so.
> Logs indicate that at some point there was not enough disk space (caused by weekend index optimization).

Software behavior becomes very difficult to define when a resource (RAM,
disk space, etc) is completely exhausted.  Even if Lucene's behavior is
well defined (which I think it might be -- the index itself is NOT
corrupt), Solr is another layer here, and I don't know whether its
behavior is well defined.  I suspect that it's not.  This might explain
what you're seeing.  That might be the only information you'll get, if
there's nothing else in the logs besides the inability to write to the disk.

Thanks,
Shawn

Re: Empty documents in Solr\lucene 3.6

[Dmitry Kan <so...@gmail.com>]

Alexey,

1. Can you take a backup of the index and run the index checker with -fix
option? Will it modify the index at all?
2. Are all the missing fields configured as stored? Are they marked as
required in the schema or optional?

Dmitry


On Tue, Apr 15, 2014 at 7:22 PM, Alexey Kozhemiakin <
Alexey_Kozhemiakin@epam.com> wrote:

> The system was up and running for long time(months) without any updates.
> There was no crashes for sure, at least support team says so.
> Logs indicate that at some point there was not enough disk space (caused
> by weekend index optimization).
>
>
> Were there any other similar cases or it's unique for us?
>
>
> Alexey.
> -----Original Message-----
> From: Shawn Heisey [mailto:solr@elyograg.org]
> Sent: Tuesday, April 15, 2014 18:50
> To: solr-user@lucene.apache.org
> Subject: Re: Empty documents in Solr\lucene 3.6
>
> Do you know for sure that the index was OK at some point?  Do you know
> what might have happened when it became not OK, like a system crash?
>
> If you have Solr logs from whatever event caused the problem, we might be
> able to figure it out ... but if you don't know when it happened or you
> don't have logs, it might not be possible to know what happened.
> The document may have simply been indexed incorrectly.
>
> Thanks,
> Shawn
>
>


-- 
Dmitry
Blog: http://dmitrykan.blogspot.com
Twitter: http://twitter.com/dmitrykan

RE: Empty documents in Solr\lucene 3.6

[Alexey Kozhemiakin <Al...@epam.com>]

The system was up and running for long time(months) without any updates.
There was no crashes for sure, at least support team says so.
Logs indicate that at some point there was not enough disk space (caused by weekend index optimization).


Were there any other similar cases or it's unique for us?


Alexey.
-----Original Message-----
From: Shawn Heisey [mailto:solr@elyograg.org] 
Sent: Tuesday, April 15, 2014 18:50
To: solr-user@lucene.apache.org
Subject: Re: Empty documents in Solr\lucene 3.6

Do you know for sure that the index was OK at some point?  Do you know what might have happened when it became not OK, like a system crash?

If you have Solr logs from whatever event caused the problem, we might be able to figure it out ... but if you don't know when it happened or you don't have logs, it might not be possible to know what happened. 
The document may have simply been indexed incorrectly.

Thanks,
Shawn

Re: Empty documents in Solr\lucene 3.6

[Shawn Heisey <so...@elyograg.org>]

On 4/15/2014 9:41 AM, Alexey Kozhemiakin wrote:
> We've faced a strange data corruption issue with one of our clients old solr setup (3.6).
>
> When we do a query (id:X OR id:Y) we get 2 nodes, one contains normal doc data, another is empty (<doc />).
> We've looked inside lucene index using Luke - same story, one of documents is empty.
> When we click on 1st document - it shows nothing.
> http://snag.gy/O5Lgq.jpg
>
>
> Probably files for stored data were corrupted? But luke index check says OK.
> Any clues how to troubleshoot root cause?

Do you know for sure that the index was OK at some point?  Do you know
what might have happened when it became not OK, like a system crash?

If you have Solr logs from whatever event caused the problem, we might
be able to figure it out ... but if you don't know when it happened or
you don't have logs, it might not be possible to know what happened. 
The document may have simply been indexed incorrectly.

Thanks,
Shawn