You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Clemens Wyss DEV <cl...@mysign.ch> on 2014/05/13 18:22:30 UTC

[lucene 4.6] NPE when calling IndexReader#openIfChanged

I am facing the following stacktrace:
	java.lang.NullPointerException: null
	  at java.io.File.<init>(File.java:305) ~[na:1.6.0_26]
	  at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.codecs.lucene40.BitVector.<init>(BitVector.java:327) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:90) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:131) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:194) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:843) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:250) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]

Unfortunately not always. Looks like j.i.File<init> is getting param child == null
What might go wrong in openIfChanged?
Thanks for any advices/hints
Clemens

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


AW: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Clemens Wyss DEV <cl...@mysign.ch>.
In this second heap dump I drilled down only one of the SegmentReaders (of the StandardDirectoryReader). Looks like StandardSegmentReader has 24 subReaders (SegmentReaders)

Class Name                                                   | Shallow Heap | Retained Heap | Percentage
---------------------------------------------------------------------------------------------------------
org.apache.lucene.index.StandardDirectoryReader @ 0x783932460|           72 |    59'255'872 |      3.04%
|- org.apache.lucene.index.SegmentReader[24] @ 0x794089ee0   |          112 |    59'190'960 |      3.03%
|  |- org.apache.lucene.index.SegmentReader @ 0x788820f40    |           72 |    16'905'072 |      0.87% <- this is the one I "drilled down"
|  |- org.apache.lucene.index.SegmentReader @ 0x783555a20    |           72 |     5'845'840 |      0.30%
|  |- org.apache.lucene.index.SegmentReader @ 0x788cea030    |           72 |     5'399'656 |      0.28%
|  |- org.apache.lucene.index.SegmentReader @ 0x7839e2bd8    |           72 |     4'256'552 |      0.22%
|  |- org.apache.lucene.index.SegmentReader @ 0x77f281b98    |           72 |     4'116'944 |      0.21%
|  |- org.apache.lucene.index.SegmentReader @ 0x7888212a0    |           72 |     3'887'368 |      0.20%
|  |- org.apache.lucene.index.SegmentReader @ 0x7845ee638    |           72 |     3'818'192 |      0.20%
|  |- org.apache.lucene.index.SegmentReader @ 0x794777c08    |           72 |     3'644'816 |      0.19%
|  |- org.apache.lucene.index.SegmentReader @ 0x783956a90    |           72 |     3'583'312 |      0.18%
|  |- org.apache.lucene.index.SegmentReader @ 0x783dd5790    |           72 |     3'153'568 |      0.16%
|  |- org.apache.lucene.index.SegmentReader @ 0x783dd57d8    |           72 |     1'017'264 |      0.05%
|  |- org.apache.lucene.index.SegmentReader @ 0x77f27b7e8    |           72 |       668'304 |      0.03%
|  |- org.apache.lucene.index.SegmentReader @ 0x77d657100    |           72 |       635'344 |      0.03%
|  |- org.apache.lucene.index.SegmentReader @ 0x77e35bb10    |           72 |       614'784 |      0.03%
|  |- org.apache.lucene.index.SegmentReader @ 0x77d676e68    |           72 |       263'176 |      0.01%
|  |- org.apache.lucene.index.SegmentReader @ 0x77d6579e8    |           72 |       244'328 |      0.01%
|  |- org.apache.lucene.index.SegmentReader @ 0x77f281b08    |           72 |       211'152 |      0.01%
|  |- org.apache.lucene.index.SegmentReader @ 0x77d6579a0    |           72 |       178'784 |      0.01%
|  |- org.apache.lucene.index.SegmentReader @ 0x77f2761b8    |           72 |       175'384 |      0.01%
|  |- org.apache.lucene.index.SegmentReader @ 0x77d657148    |           72 |       167'128 |      0.01%
|  |- org.apache.lucene.index.SegmentReader @ 0x77d676e20    |           72 |       164'128 |      0.01%
|  |- org.apache.lucene.index.SegmentReader @ 0x77d657a30    |           72 |       150'560 |      0.01%
|  |- org.apache.lucene.index.SegmentReader @ 0x77d676eb0    |           72 |        53'992 |      0.00%
|  |- org.apache.lucene.index.SegmentReader @ 0x77d676ef8    |           72 |        35'200 |      0.00%
|  '- Total: 24 entries                                      |              |               |           
---------------------------------------------------------------------------------------------------------

-----Ursprüngliche Nachricht-----
Von: Michael McCandless [mailto:lucene@mikemccandless.com] 
Gesendet: Montag, 16. Juni 2014 10:59
An: Lucene Users
Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Wait, in fst.ByteStore I see "only" 5'485'824 -- does this mean
5485824 bytes, or ~5.2 MB?  This is probably "correct", meaning this is the RAM to hold the terms index.  But I can't see from your heap dump output where the other ~51.3 MB is being used by StandardDirectoryReader.

Mike McCandless

http://blog.mikemccandless.com


On Sun, Jun 15, 2014 at 2:16 PM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>> bytes (ie, ~56.5 MB) being used by the StandardDirectoryReader?
> Right, namely in org.apache.lucene.util.fst.BytesStore's
>
> Class Name                                                                                        | Shallow Heap | Retained Heap | Percentage
> ----------------------------------------------------------------------------------------------------------------------------------------------
> org.apache.lucene.index.StandardDirectoryReader @ 0x783932460                                     |           72 |    59'255'872 |      3.04%
> |- org.apache.lucene.index.SegmentReader[24] @ 0x794089ee0                                        |          112 |    59'190'960 |      3.03%
> |  |- org.apache.lucene.index.SegmentReader @ 0x788820f40                                         |           72 |    16'905'072 |      0.87%
> |  |  |- org.apache.lucene.index.SegmentCoreReaders @ 0x7910cacc8                                 |           56 |    16'895'576 |      0.87%
> |  |  |  |- org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader @ 0x780661c50   |           24 |    16'864'864 |      0.86%
> |  |  |  |  |- org.apache.lucene.codecs.BlockTreeTermsReader @ 0x7910cae50                        |           56 |    16'864'240 |      0.86%
> |  |  |  |  |  |- java.util.TreeMap @ 0x783902738                                                 |           48 |    16'858'472 |      0.86%
> |  |  |  |  |  |  '- java.util.TreeMap$Entry @ 0x77ec5f9f8                                        |           40 |    16'858'424 |      0.86%
> |  |  |  |  |  |     |- java.util.TreeMap$Entry @ 0x77ec5fa20                                     |           40 |    10'895'656 |      0.56%
> |  |  |  |  |  |     |  |- org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader @ 0x78347fc08|           72 |     5'497'328 |      0.28%
> |  |  |  |  |  |     |  |  |- org.apache.lucene.util.fst.FST @ 0x788fe36d0                        |          104 |     5'497'208 |      0.28%
> |  |  |  |  |  |     |  |  |  |- org.apache.lucene.util.fst.BytesStore @ 0x77ec5fe80              |           40 |     5'485'824 |      0.28%
> |  |  |  |  |  |     |  |  |  |- org.apache.lucene.util.fst.FST$Arc[128] @ 0x786efec80            |          528 |        11'216 |      0.00%
> |  |  |  |  |  |     |  |  |  |- org.apache.lucene.util.BytesRef @ 0x780664410                    |           24 |            48 |      0.00%
> |  |  |  |  |  |     |  |  |  |- int[0] @ 0x7806643f8                                             |           16 |            16 |      0.00%
> |  |  |  |  |  |     |  |  |  '- Total: 4 entries                                                 |              |               |
> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> --
> -----Ursprüngliche Nachricht-----
> Von: Michael McCandless [mailto:lucene@mikemccandless.com]
> Gesendet: Freitag, 13. Juni 2014 15:48
> An: Lucene Users
> Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>
> On Fri, Jun 13, 2014 at 8:53 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>> Thanks a lot!
>>>"large text fields"
>> What is a good limit (in characters) to switch from StringField to TextField? Do <Langugae>Analyzers (e.g. GermanAnalyzer)  help a lot in reducing the size of an Index?
>
> It's more based on your app's requirements.  StringField indexes everything as a single token.
>
>>> Add XXXDocValuesField instead of e.g. StringField.
>> Does this apply only for StringFields? Or for TextFields too?
>>
>>> Upgrade to the upcoming Lucene 4.9
>> we have not yet transitionen to Java 7/8 ... hopefully soon ;)
>>
>>> and take a heap dump and see what's using RAM
>> Find attached a snippet from MemoryAnalyzer
>
> Does this say  59255872 bytes (ie, ~56.5 MB) being used by the StandardDirectoryReader?
>
> I'm a little confused because I don't see which structures sum up to that total.  And I would expect the FST (terms index) to take more RAM.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Michael McCandless <lu...@mikemccandless.com>.
Wait, in fst.ByteStore I see "only" 5'485'824 -- does this mean
5485824 bytes, or ~5.2 MB?  This is probably "correct", meaning this
is the RAM to hold the terms index.  But I can't see from your heap
dump output where the other ~51.3 MB is being used by
StandardDirectoryReader.

Mike McCandless

http://blog.mikemccandless.com


On Sun, Jun 15, 2014 at 2:16 PM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>> bytes (ie, ~56.5 MB) being used by the StandardDirectoryReader?
> Right, namely in org.apache.lucene.util.fst.BytesStore's
>
> Class Name                                                                                        | Shallow Heap | Retained Heap | Percentage
> ----------------------------------------------------------------------------------------------------------------------------------------------
> org.apache.lucene.index.StandardDirectoryReader @ 0x783932460                                     |           72 |    59'255'872 |      3.04%
> |- org.apache.lucene.index.SegmentReader[24] @ 0x794089ee0                                        |          112 |    59'190'960 |      3.03%
> |  |- org.apache.lucene.index.SegmentReader @ 0x788820f40                                         |           72 |    16'905'072 |      0.87%
> |  |  |- org.apache.lucene.index.SegmentCoreReaders @ 0x7910cacc8                                 |           56 |    16'895'576 |      0.87%
> |  |  |  |- org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader @ 0x780661c50   |           24 |    16'864'864 |      0.86%
> |  |  |  |  |- org.apache.lucene.codecs.BlockTreeTermsReader @ 0x7910cae50                        |           56 |    16'864'240 |      0.86%
> |  |  |  |  |  |- java.util.TreeMap @ 0x783902738                                                 |           48 |    16'858'472 |      0.86%
> |  |  |  |  |  |  '- java.util.TreeMap$Entry @ 0x77ec5f9f8                                        |           40 |    16'858'424 |      0.86%
> |  |  |  |  |  |     |- java.util.TreeMap$Entry @ 0x77ec5fa20                                     |           40 |    10'895'656 |      0.56%
> |  |  |  |  |  |     |  |- org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader @ 0x78347fc08|           72 |     5'497'328 |      0.28%
> |  |  |  |  |  |     |  |  |- org.apache.lucene.util.fst.FST @ 0x788fe36d0                        |          104 |     5'497'208 |      0.28%
> |  |  |  |  |  |     |  |  |  |- org.apache.lucene.util.fst.BytesStore @ 0x77ec5fe80              |           40 |     5'485'824 |      0.28%
> |  |  |  |  |  |     |  |  |  |- org.apache.lucene.util.fst.FST$Arc[128] @ 0x786efec80            |          528 |        11'216 |      0.00%
> |  |  |  |  |  |     |  |  |  |- org.apache.lucene.util.BytesRef @ 0x780664410                    |           24 |            48 |      0.00%
> |  |  |  |  |  |     |  |  |  |- int[0] @ 0x7806643f8                                             |           16 |            16 |      0.00%
> |  |  |  |  |  |     |  |  |  '- Total: 4 entries                                                 |              |               |
> ----------------------------------------------------------------------------------------------------------------------------------------------
> -----Ursprüngliche Nachricht-----
> Von: Michael McCandless [mailto:lucene@mikemccandless.com]
> Gesendet: Freitag, 13. Juni 2014 15:48
> An: Lucene Users
> Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>
> On Fri, Jun 13, 2014 at 8:53 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>> Thanks a lot!
>>>"large text fields"
>> What is a good limit (in characters) to switch from StringField to TextField? Do <Langugae>Analyzers (e.g. GermanAnalyzer)  help a lot in reducing the size of an Index?
>
> It's more based on your app's requirements.  StringField indexes everything as a single token.
>
>>> Add XXXDocValuesField instead of e.g. StringField.
>> Does this apply only for StringFields? Or for TextFields too?
>>
>>> Upgrade to the upcoming Lucene 4.9
>> we have not yet transitionen to Java 7/8 ... hopefully soon ;)
>>
>>> and take a heap dump and see what's using RAM
>> Find attached a snippet from MemoryAnalyzer
>
> Does this say  59255872 bytes (ie, ~56.5 MB) being used by the StandardDirectoryReader?
>
> I'm a little confused because I don't see which structures sum up to that total.  And I would expect the FST (terms index) to take more RAM.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


AW: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Clemens Wyss DEV <cl...@mysign.ch>.
> bytes (ie, ~56.5 MB) being used by the StandardDirectoryReader?
Right, namely in org.apache.lucene.util.fst.BytesStore's 

Class Name                                                                                        | Shallow Heap | Retained Heap | Percentage
----------------------------------------------------------------------------------------------------------------------------------------------
org.apache.lucene.index.StandardDirectoryReader @ 0x783932460                                     |           72 |    59'255'872 |      3.04%
|- org.apache.lucene.index.SegmentReader[24] @ 0x794089ee0                                        |          112 |    59'190'960 |      3.03%
|  |- org.apache.lucene.index.SegmentReader @ 0x788820f40                                         |           72 |    16'905'072 |      0.87%
|  |  |- org.apache.lucene.index.SegmentCoreReaders @ 0x7910cacc8                                 |           56 |    16'895'576 |      0.87%
|  |  |  |- org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader @ 0x780661c50   |           24 |    16'864'864 |      0.86%
|  |  |  |  |- org.apache.lucene.codecs.BlockTreeTermsReader @ 0x7910cae50                        |           56 |    16'864'240 |      0.86%
|  |  |  |  |  |- java.util.TreeMap @ 0x783902738                                                 |           48 |    16'858'472 |      0.86%
|  |  |  |  |  |  '- java.util.TreeMap$Entry @ 0x77ec5f9f8                                        |           40 |    16'858'424 |      0.86%
|  |  |  |  |  |     |- java.util.TreeMap$Entry @ 0x77ec5fa20                                     |           40 |    10'895'656 |      0.56%
|  |  |  |  |  |     |  |- org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader @ 0x78347fc08|           72 |     5'497'328 |      0.28%
|  |  |  |  |  |     |  |  |- org.apache.lucene.util.fst.FST @ 0x788fe36d0                        |          104 |     5'497'208 |      0.28%
|  |  |  |  |  |     |  |  |  |- org.apache.lucene.util.fst.BytesStore @ 0x77ec5fe80              |           40 |     5'485'824 |      0.28%
|  |  |  |  |  |     |  |  |  |- org.apache.lucene.util.fst.FST$Arc[128] @ 0x786efec80            |          528 |        11'216 |      0.00%
|  |  |  |  |  |     |  |  |  |- org.apache.lucene.util.BytesRef @ 0x780664410                    |           24 |            48 |      0.00%
|  |  |  |  |  |     |  |  |  |- int[0] @ 0x7806643f8                                             |           16 |            16 |      0.00%
|  |  |  |  |  |     |  |  |  '- Total: 4 entries                                                 |              |               |           
----------------------------------------------------------------------------------------------------------------------------------------------
-----Ursprüngliche Nachricht-----
Von: Michael McCandless [mailto:lucene@mikemccandless.com] 
Gesendet: Freitag, 13. Juni 2014 15:48
An: Lucene Users
Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

On Fri, Jun 13, 2014 at 8:53 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
> Thanks a lot!
>>"large text fields"
> What is a good limit (in characters) to switch from StringField to TextField? Do <Langugae>Analyzers (e.g. GermanAnalyzer)  help a lot in reducing the size of an Index?

It's more based on your app's requirements.  StringField indexes everything as a single token.

>> Add XXXDocValuesField instead of e.g. StringField.
> Does this apply only for StringFields? Or for TextFields too?
>
>> Upgrade to the upcoming Lucene 4.9
> we have not yet transitionen to Java 7/8 ... hopefully soon ;)
>
>> and take a heap dump and see what's using RAM
> Find attached a snippet from MemoryAnalyzer

Does this say  59255872 bytes (ie, ~56.5 MB) being used by the StandardDirectoryReader?

I'm a little confused because I don't see which structures sum up to that total.  And I would expect the FST (terms index) to take more RAM.

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Michael McCandless <lu...@mikemccandless.com>.
On Fri, Jun 13, 2014 at 8:53 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
> Thanks a lot!
>>"large text fields"
> What is a good limit (in characters) to switch from StringField to TextField? Do <Langugae>Analyzers (e.g. GermanAnalyzer)  help a lot in reducing the size of an Index?

It's more based on your app's requirements.  StringField indexes
everything as a single token.

>> Add XXXDocValuesField instead of e.g. StringField.
> Does this apply only for StringFields? Or for TextFields too?
>
>> Upgrade to the upcoming Lucene 4.9
> we have not yet transitionen to Java 7/8 ... hopefully soon ;)
>
>> and take a heap dump and see what's using RAM
> Find attached a snippet from MemoryAnalyzer

Does this say  59255872 bytes (ie, ~56.5 MB) being used by the
StandardDirectoryReader?

I'm a little confused because I don't see which structures sum up to
that total.  And I would expect the FST (terms index) to take more
RAM.

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


AW: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Clemens Wyss DEV <cl...@mysign.ch>.
Thanks a lot!
>"large text fields"
What is a good limit (in characters) to switch from StringField to TextField? Do <Langugae>Analyzers (e.g. GermanAnalyzer)  help a lot in reducing the size of an Index?

> Add XXXDocValuesField instead of e.g. StringField.
Does this apply only for StringFields? Or for TextFields too?

> Upgrade to the upcoming Lucene 4.9
we have not yet transitionen to Java 7/8 ... hopefully soon ;)

> and take a heap dump and see what's using RAM
Find attached a snippet from MemoryAnalyzer
Class Name                                                                                                                                         | Shallow Heap | Retained Heap | Percentage
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
org.apache.lucene.index.StandardDirectoryReader @ 0x783932460                                                                                      |           72 |    59'255'872 |      3.04%
|- org.apache.lucene.index.SegmentReader[24] @ 0x794089ee0                                                                                         |          112 |    59'190'960 |      3.03%
|  |- org.apache.lucene.index.SegmentReader @ 0x788820f40                                                                                          |           72 |    16'905'072 |      0.87%
|  |  |- org.apache.lucene.index.SegmentCoreReaders @ 0x7910cacc8                                                                                  |           56 |    16'895'576 |      0.87%
|  |  |  |- org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader @ 0x780661c50                                                    |           24 |    16'864'864 |      0.86%
|  |  |  |  |- org.apache.lucene.codecs.BlockTreeTermsReader @ 0x7910cae50                                                                         |           56 |    16'864'240 |      0.86%
|  |  |  |  |  |- java.util.TreeMap @ 0x783902738                                                                                                  |           48 |    16'858'472 |      0.86%
|  |  |  |  |  |  '- java.util.TreeMap$Entry @ 0x77ec5f9f8                                                                                         |           40 |    16'858'424 |      0.86%
|  |  |  |  |  |     |- java.util.TreeMap$Entry @ 0x77ec5fa20                                                                                      |           40 |    10'895'656 |      0.56%
|  |  |  |  |  |     |- java.util.TreeMap$Entry @ 0x77ec5fa48                                                                                      |           40 |     5'960'072 |      0.31%
|  |  |  |  |  |     |  |- java.util.TreeMap$Entry @ 0x77ec5fa98                                                                                   |           40 |     5'958'072 |      0.31%
|  |  |  |  |  |     |  |  |- java.util.TreeMap$Entry @ 0x77fc09bf0                                                                                |           40 |     5'949'864 |      0.30%
|  |  |  |  |  |     |  |  |- org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader @ 0x788820e20                                              |           72 |         8'168 |      0.00%
|  |  |  |  |  |     |  |  '- Total: 2 entries                                                                                                     |              |               |           
|  |  |  |  |  |     |  |- java.util.TreeMap$Entry @ 0x77ec5fa70                                                                                   |           40 |         1'000 |      0.00%
|  |  |  |  |  |     |  |  '- org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader @ 0x78347fbc0                                              |           72 |           960 |      0.00%
|  |  |  |  |  |     |  |     |- org.apache.lucene.util.fst.FST @ 0x788fe34c8                                                                      |          104 |           840 |      0.00%
|  |  |  |  |  |     |  |     |  |- org.apache.lucene.util.fst.FST$Arc[128] @ 0x7870932a0                                                          |          528 |           528 |      0.00%
|  |  |  |  |  |     |  |     |  |- org.apache.lucene.util.fst.BytesStore @ 0x77ec5fb60                                                            |           40 |           144 |      0.00%
|  |  |  |  |  |     |  |     |  |  '- java.util.ArrayList @ 0x780663b28                                                                           |           24 |           104 |      0.00%
|  |  |  |  |  |     |  |     |  |- org.apache.lucene.util.BytesRef @ 0x780663b10                                                                  |           24 |            48 |      0.00%
|  |  |  |  |  |     |  |     |  |  '- byte[5] @ 0x780663b58  .....                                                                                |           24 |            24 |      0.00%
|  |  |  |  |  |     |  |     |  |- int[0] @ 0x780663af8                                                                                           |           16 |            16 |      0.00%
|  |  |  |  |  |     |  |     |  '- Total: 4 entries                                                                                               |              |               |           
|  |  |  |  |  |     |  |     |- org.apache.lucene.util.BytesRef @ 0x780663ae0                                                                     |           24 |            48 |      0.00%
|  |  |  |  |  |     |  |     '- Total: 2 entries                                                                                                  |              |               |           
|  |  |  |  |  |     |  |- org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader @ 0x788820dd8                                                 |           72 |           960 |      0.00%
|  |  |  |  |  |     |  '- Total: 3 entries                                                                                                        |              |               |           
|  |  |  |  |  |     |- org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader @ 0x788820d90                                                    |           72 |         2'656 |      0.00%
|  |  |  |  |  |     '- Total: 3 entries                                                                                                           |              |               |           
|  |  |  |  |  |- org.apache.lucene.codecs.lucene41.Lucene41PostingsReader @ 0x78274ab88                                                           |           32 |         4'032 |      0.00%
|  |  |  |  |  |- org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput @ 0x788820d48                                                             |           72 |         1'680 |      0.00%
|  |  |  |  |  '- Total: 3 entries                                                                                                                 |              |               |           
|  |  |  |  |- java.util.TreeMap @ 0x783902798                                                                                                     |           48 |           368 |      0.00%
|  |  |  |  |- java.util.HashMap @ 0x7839027c8                                                                                                     |           48 |           232 |      0.00%
|  |  |  |  '- Total: 3 entries                                                                                                                    |              |               |           
|  |  |  |- org.apache.lucene.index.SegmentCoreReaders$1 @ 0x78274aaa8                                                                             |           32 |        17'688 |      0.00%
|  |  |  |- org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer @ 0x7822983c0                                                              |           48 |         6'504 |      0.00%
|  |  |  |- org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer$3 @ 0x7b1424f10                                                            |           24 |         3'456 |      0.00%
|  |  |  |- org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader @ 0x7910e98c8                                                       |           56 |         1'240 |      0.00%
|  |  |  |- org.apache.lucene.index.SegmentCoreReaders$3 @ 0x78274aae8                                                                             |           32 |           456 |      0.00%
|  |  |  |- org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexReader @ 0x77fb743a0                                                  |           40 |           344 |      0.00%
|  |  |  |- java.lang.String @ 0x78292d4c8  NIOFSIndexInput(path="/opt/webs/fust.ch/WEB-INF/indexes/1/fr_CH_1/fustusermanuals/full/__data/_n8.fdt")|           32 |           256 |      0.00%
|  |  |  |- org.apache.lucene.index.SegmentCoreReaders$2 @ 0x78274aac8                                                                             |           32 |           240 |      0.00%
|  |  |  |- java.util.Collections$SynchronizedSet @ 0x780661c68                                                                                    |           24 |           216 |      0.00%
|  |  |  |- sun.nio.ch.FileChannelImpl @ 0x782298420                                                                                               |           48 |           152 |      0.00%
|  |  |  |- java.io.RandomAccessFile @ 0x782933780                                                                                                 |           32 |            48 |      0.00%
|  |  |  |- java.io.FileDescriptor @ 0x780b56148                                                                                                   |           24 |            40 |      0.00%
|  |  |  |- java.util.concurrent.atomic.AtomicInteger @ 0x780661c38                                                                                |           16 |            16 |      0.00%
|  |  |  '- Total: 14 entries                                                                                                                      |              |               |           
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 
Does this help?

-----Ursprüngliche Nachricht-----
Von: Michael McCandless [mailto:lucene@mikemccandless.com] 
Gesendet: Freitag, 13. Juni 2014 13:15
An: Lucene Users
Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

On Fri, Jun 13, 2014 at 3:02 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>> limit how many fields have norms enabled
> We have one index for approx 7000 pdfs (24GB). Of course no content is STOREd (but ANALYZEd). This very index occupies 4GB on disk and the corresponding IndexReader is 60MB.
> Are norms per default enabled org.apache.lucene.document .TextField?

Yes.  Norms are a good idea for "large text fields", e.g. body text or a catch all field, but usually not a good idea for tiny fields (e.g.
title).

>> use disk-based doc values not field cache
> How is this done?

Add XXXDocValuesField instead of e.g. StringField.

>> etc.
> such as? ;)

Upgrade to the upcoming Lucene 4.9; there have been some improvements e.g. to norms compression.  You can tune your terms index settings, but terms index usually doesn't use much RAM.

You can fire up your up, get all searchers warmed, and take a heap dump and see what's using RAM.  We can iterate from there.

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Michael McCandless <lu...@mikemccandless.com>.
On Fri, Jun 13, 2014 at 3:02 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>> limit how many fields have norms enabled
> We have one index for approx 7000 pdfs (24GB). Of course no content is STOREd (but ANALYZEd). This very index occupies 4GB on disk and the corresponding IndexReader is 60MB.
> Are norms per default enabled org.apache.lucene.document .TextField?

Yes.  Norms are a good idea for "large text fields", e.g. body text or
a catch all field, but usually not a good idea for tiny fields (e.g.
title).

>> use disk-based doc values not field cache
> How is this done?

Add XXXDocValuesField instead of e.g. StringField.

>> etc.
> such as? ;)

Upgrade to the upcoming Lucene 4.9; there have been some improvements
e.g. to norms compression.  You can tune your terms index settings,
but terms index usually doesn't use much RAM.

You can fire up your up, get all searchers warmed, and take a heap
dump and see what's using RAM.  We can iterate from there.

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


AW: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Clemens Wyss DEV <cl...@mysign.ch>.
> limit how many fields have norms enabled
We have one index for approx 7000 pdfs (24GB). Of course no content is STOREd (but ANALYZEd). This very index occupies 4GB on disk and the corresponding IndexReader is 60MB.
Are norms per default enabled org.apache.lucene.document .TextField? 

> use disk-based doc values not field cache
How is this done?

> etc.
such as? ;)

-----Ursprüngliche Nachricht-----
Von: Michael McCandless [mailto:lucene@mikemccandless.com] 
Gesendet: Mittwoch, 21. Mai 2014 11:21
An: Lucene Users
Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

On Wed, May 21, 2014 at 3:17 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>> Can you just decrease IW's ramBufferSizeMB to relieve the memory pressure?
> +1
> Is there something alike for IndexReaders?

No, although you can take steps during indexing to reduce the RAM required during searching, e.g. limit how many fields have norms enabled, use disk-based doc values not field cache, etc.

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Michael McCandless <lu...@mikemccandless.com>.
On Wed, May 21, 2014 at 3:17 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>> Can you just decrease IW's ramBufferSizeMB to relieve the memory pressure?
> +1
> Is there something alike for IndexReaders?

No, although you can take steps during indexing to reduce the RAM
required during searching, e.g. limit how many fields have norms
enabled, use disk-based doc values not field cache, etc.

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


AW: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Clemens Wyss DEV <cl...@mysign.ch>.
> Can you just decrease IW's ramBufferSizeMB to relieve the memory pressure?
+1
Is there something alike for IndexReaders?

-----Ursprüngliche Nachricht-----
Von: Michael McCandless [mailto:lucene@mikemccandless.com] 
Gesendet: Montag, 19. Mai 2014 12:19
An: Lucene Users
Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

On Mon, May 19, 2014 at 6:14 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
> Mike,
> first of all thanks for all your input, I really appreciate (as much as I like reading your blog).

You're welcome!

>> Hmm, but you swap these files over while an IndexReader is still open on the index?
> no IndexReader is open while swapping. At least not by design. We have 
> at most one (current)reader per Index, which is closed just before 
> swapping. I will investigate on this more deeply, though, me not being 
> impeccable ;)

Hmm, double check that.  The only way I can explain what you're seeing is if the reader was open on the index, then the index was blown away / replaced, then openIfChanged called ...

>> Instead, use IndexWriter.deleteAll, then index all your new docs, 
>>then commit, then reopen the reader.
> we were facing memory issues and hence decided to do "intermediate commits" (i.e. flush the writer). And in order to still provide "atomicity" (with respect to a full re-index) we decided to go the before mentionend approach.

Can you just decrease IW's ramBufferSizeMB to relieve the memory pressure?

>> Separately, it's better to use a near-real-time IndexReader so you 
>>don't have to commit in order to see changes.
> What if I were to search right after deleteAll?

That's fine, it's fully transactional: the deleteAll will not be visible until you open a new reader.  If you change your mind, you can call rollback after deleteAll and the docs are back, etc.

Even opening a new IW with OpenMode.CREATE is transactional: that change will not become visible to a reader until it's reopened.

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Michael McCandless <lu...@mikemccandless.com>.
On Mon, May 19, 2014 at 6:14 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
> Mike,
> first of all thanks for all your input, I really appreciate (as much as I like reading your blog).

You're welcome!

>> Hmm, but you swap these files over while an IndexReader is still open on the index?
> no IndexReader is open while swapping. At least not by design. We have at most one (current)reader per Index, which is closed just before swapping. I will investigate on this more deeply, though, me not being impeccable ;)

Hmm, double check that.  The only way I can explain what you're seeing
is if the reader was open on the index, then the index was blown away
/ replaced, then openIfChanged called ...

>> Instead, use IndexWriter.deleteAll, then index all your new docs,
>>then commit, then reopen the reader.
> we were facing memory issues and hence decided to do "intermediate commits" (i.e. flush the writer). And in order to still provide "atomicity" (with respect to a full re-index) we decided to go the before mentionend approach.

Can you just decrease IW's ramBufferSizeMB to relieve the memory pressure?

>> Separately, it's better to use a near-real-time IndexReader so
>>you don't have to commit in order to see changes.
> What if I were to search right after deleteAll?

That's fine, it's fully transactional: the deleteAll will not be
visible until you open a new reader.  If you change your mind, you can
call rollback after deleteAll and the docs are back, etc.

Even opening a new IW with OpenMode.CREATE is transactional: that
change will not become visible to a reader until it's reopened.

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


AW: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Clemens Wyss DEV <cl...@mysign.ch>.
Mike,
first of all thanks for all your input, I really appreciate (as much as I like reading your blog).

> Hmm, but you swap these files over while an IndexReader is still open on the index?
no IndexReader is open while swapping. At least not by design. We have at most one (current)reader per Index, which is closed just before swapping. I will investigate on this more deeply, though, me not being impeccable ;)

> Instead, use IndexWriter.deleteAll, then index all your new docs, 
>then commit, then reopen the reader.
we were facing memory issues and hence decided to do "intermediate commits" (i.e. flush the writer). And in order to still provide "atomicity" (with respect to a full re-index) we decided to go the before mentionend approach.

> Separately, it's better to use a near-real-time IndexReader so 
>you don't have to commit in order to see changes.
What if I were to search right after deleteAll?


-----Ursprüngliche Nachricht-----
Von: Michael McCandless [mailto:lucene@mikemccandless.com] 
Gesendet: Montag, 19. Mai 2014 11:05
An: Lucene Users
Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

On Mon, May 19, 2014 at 4:59 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>> Are you using doc-values updates?
> Not to my knowledge, i.e. not explicitly

Hmm ok.

>> Are you ever removing files directly from the index directory yourself between reopens?
> Yes. Reindexing an index completely(*) is done in a separate temporary index/folder. After that we (guarded by a mutex) swap the files, i.e. delete the old ones and then move over the new index files.
>
> (*)we do this for example after a massimport of index-relevant data (instead of updating each and every element on ist own).

Hmm, but you swap these files over while an IndexReader is still open on the index?  And then you call openIfChanged?

If so, that's what's leading to this exception: you shouldn't ever need to overwrite files at index level.

Instead, use IndexWriter.deleteAll, then index all your new docs, then commit, then reopen the reader.  Alternatively, open IndexWriter with OpenMode.CREATE on the index, index all your docs, commit, and reopen.

Separately, it's better to use a near-real-time IndexReader so you don't have to commit in order to see changes.

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Michael McCandless <lu...@mikemccandless.com>.
On Mon, May 19, 2014 at 4:59 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>> Are you using doc-values updates?
> Not to my knowledge, i.e. not explicitly

Hmm ok.

>> Are you ever removing files directly from the index directory yourself between reopens?
> Yes. Reindexing an index completely(*) is done in a separate temporary index/folder. After that we (guarded by a mutex) swap the files, i.e. delete the old ones and then move over the new index files.
>
> (*)we do this for example after a massimport of index-relevant data (instead of updating each and every element on ist own).

Hmm, but you swap these files over while an IndexReader is still open
on the index?  And then you call openIfChanged?

If so, that's what's leading to this exception: you shouldn't ever
need to overwrite files at index level.

Instead, use IndexWriter.deleteAll, then index all your new docs, then
commit, then reopen the reader.  Alternatively, open IndexWriter with
OpenMode.CREATE on the index, index all your docs, commit, and reopen.

Separately, it's better to use a near-real-time IndexReader so you
don't have to commit in order to see changes.

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


AW: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Clemens Wyss DEV <cl...@mysign.ch>.
> Are you using doc-values updates?
Not to my knowledge, i.e. not explicitly

> Are you ever removing files directly from the index directory yourself between reopens?
Yes. Reindexing an index completely(*) is done in a separate temporary index/folder. After that we (guarded by a mutex) swap the files, i.e. delete the old ones and then move over the new index files.

(*)we do this for example after a massimport of index-relevant data (instead of updating each and every element on ist own).

-----Ursprüngliche Nachricht-----
Von: Michael McCandless [mailto:lucene@mikemccandless.com] 
Gesendet: Montag, 19. Mai 2014 10:44
An: Lucene Users
Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Hmm, I was wrong before, the code is more complex than I thought.

Are you using doc-values updates?

Can you describe a bit more how your app works?  Are you ever removing files directly from the index directory yourself between reopens?

Mike McCandless

http://blog.mikemccandless.com


On Mon, May 19, 2014 at 1:36 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
> Sorry for being imprecise
> java version "1.6.0_26"
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 
> 64-Bit Server VM (build 20.1-b02, mixed mode)
>
> - Clemens
> -----Ursprüngliche Nachricht-----
> Von: Michael McCandless [mailto:lucene@mikemccandless.com]
> Gesendet: Sonntag, 18. Mai 2014 16:51
> An: Lucene Users
> Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>
> But what is the output of "java -fullversion"?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sun, May 18, 2014 at 5:24 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>>> What java version?
>> We are still on (1.)6
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Michael McCandless [mailto:lucene@mikemccandless.com]
>> Gesendet: Donnerstag, 15. Mai 2014 11:43
>> An: Lucene Users
>> Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>>
>> delGen=-1 means there are no deletions, but the exception makes no 
>> sense because up above SegmentReader.java calls si.hasDeletions() 
>> which returns delGen != -1 which should have mean 
>> Lucene40LiveDocsFormat.readLiveDocs should not have been called.  It 
>> seems impossible :)
>>
>> What java version?
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Wed, May 14, 2014 at 2:16 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>>> Tackled this down a little bit more:
>>> Lucene40LiveDocsFormat#readLiveDocs calls 
>>> IndexFileNames#fileNameForGeneration
>>> If I get this right, param 'gen' seems to be -1.
>>> Gen is being gathered from SegmentCommitInfo#getDelGen
>>>
>>> What does "delGen == -1" mean?
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Clemens Wyss DEV [mailto:clemensdev@mysign.ch]
>>> Gesendet: Dienstag, 13. Mai 2014 18:23
>>> An: java-user@lucene.apache.org
>>> Betreff: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>>>
>>> I am facing the following stacktrace:
>>>         java.lang.NullPointerException: null
>>>           at java.io.File.<init>(File.java:305) ~[na:1.6.0_26]
>>>           at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.codecs.lucene40.BitVector.<init>(BitVector.java:327) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:90) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:131) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:194) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:843) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:250) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at
>>> org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.
>>> java:170) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 
>>> 11:05:50]
>>>
>>> Unfortunately not always. Looks like j.i.File<init> is getting param child == null What might go wrong in openIfChanged?
>>> Thanks for any advices/hints
>>> Clemens
>>>
>>> --------------------------------------------------------------------
>>> - To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>> --------------------------------------------------------------------
>>> - To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Michael McCandless <lu...@mikemccandless.com>.
Hmm, I was wrong before, the code is more complex than I thought.

Are you using doc-values updates?

Can you describe a bit more how your app works?  Are you ever removing
files directly from the index directory yourself between reopens?

Mike McCandless

http://blog.mikemccandless.com


On Mon, May 19, 2014 at 1:36 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
> Sorry for being imprecise
> java version "1.6.0_26"
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
>
> - Clemens
> -----Ursprüngliche Nachricht-----
> Von: Michael McCandless [mailto:lucene@mikemccandless.com]
> Gesendet: Sonntag, 18. Mai 2014 16:51
> An: Lucene Users
> Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>
> But what is the output of "java -fullversion"?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sun, May 18, 2014 at 5:24 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>>> What java version?
>> We are still on (1.)6
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Michael McCandless [mailto:lucene@mikemccandless.com]
>> Gesendet: Donnerstag, 15. Mai 2014 11:43
>> An: Lucene Users
>> Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>>
>> delGen=-1 means there are no deletions, but the exception makes no
>> sense because up above SegmentReader.java calls si.hasDeletions()
>> which returns delGen != -1 which should have mean
>> Lucene40LiveDocsFormat.readLiveDocs should not have been called.  It
>> seems impossible :)
>>
>> What java version?
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Wed, May 14, 2014 at 2:16 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>>> Tackled this down a little bit more:
>>> Lucene40LiveDocsFormat#readLiveDocs calls
>>> IndexFileNames#fileNameForGeneration
>>> If I get this right, param 'gen' seems to be -1.
>>> Gen is being gathered from SegmentCommitInfo#getDelGen
>>>
>>> What does "delGen == -1" mean?
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Clemens Wyss DEV [mailto:clemensdev@mysign.ch]
>>> Gesendet: Dienstag, 13. Mai 2014 18:23
>>> An: java-user@lucene.apache.org
>>> Betreff: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>>>
>>> I am facing the following stacktrace:
>>>         java.lang.NullPointerException: null
>>>           at java.io.File.<init>(File.java:305) ~[na:1.6.0_26]
>>>           at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.codecs.lucene40.BitVector.<init>(BitVector.java:327) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:90) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:131) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:194) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:843) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:250) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>>           at
>>> org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.
>>> java:170) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19
>>> 11:05:50]
>>>
>>> Unfortunately not always. Looks like j.i.File<init> is getting param child == null What might go wrong in openIfChanged?
>>> Thanks for any advices/hints
>>> Clemens
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


AW: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Clemens Wyss DEV <cl...@mysign.ch>.
> delGen=-1 means there are no deletions, but the exception makes no 
> sense because up above SegmentReader.java calls si.hasDeletions() 
> which returns delGen != -1 which should have mean 
> Lucene40LiveDocsFormat.readLiveDocs should not have been called.  It 
> seems impossible :)
Possibly a concurrency/timing issue?

-----Ursprüngliche Nachricht-----
Von: Clemens Wyss DEV [mailto:clemensdev@mysign.ch] 
Gesendet: Montag, 19. Mai 2014 07:37
An: java-user@lucene.apache.org
Betreff: AW: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Sorry for being imprecise
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

- Clemens
-----Ursprüngliche Nachricht-----
Von: Michael McCandless [mailto:lucene@mikemccandless.com]
Gesendet: Sonntag, 18. Mai 2014 16:51
An: Lucene Users
Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

But what is the output of "java -fullversion"?

Mike McCandless

http://blog.mikemccandless.com

On Sun, May 18, 2014 at 5:24 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>> What java version?
> We are still on (1.)6
>
> -----Ursprüngliche Nachricht-----
> Von: Michael McCandless [mailto:lucene@mikemccandless.com]
> Gesendet: Donnerstag, 15. Mai 2014 11:43
> An: Lucene Users
> Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>
> delGen=-1 means there are no deletions, but the exception makes no 
> sense because up above SegmentReader.java calls si.hasDeletions() 
> which returns delGen != -1 which should have mean 
> Lucene40LiveDocsFormat.readLiveDocs should not have been called.  It 
> seems impossible :)
>
> What java version?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Wed, May 14, 2014 at 2:16 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>> Tackled this down a little bit more:
>> Lucene40LiveDocsFormat#readLiveDocs calls 
>> IndexFileNames#fileNameForGeneration
>> If I get this right, param 'gen' seems to be -1.
>> Gen is being gathered from SegmentCommitInfo#getDelGen
>>
>> What does "delGen == -1" mean?
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Clemens Wyss DEV [mailto:clemensdev@mysign.ch]
>> Gesendet: Dienstag, 13. Mai 2014 18:23
>> An: java-user@lucene.apache.org
>> Betreff: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>>
>> I am facing the following stacktrace:
>>         java.lang.NullPointerException: null
>>           at java.io.File.<init>(File.java:305) ~[na:1.6.0_26]
>>           at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.codecs.lucene40.BitVector.<init>(BitVector.java:327) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:90) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:131) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:194) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:843) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:250) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at
>> org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.
>> java:170) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 
>> 11:05:50]
>>
>> Unfortunately not always. Looks like j.i.File<init> is getting param child == null What might go wrong in openIfChanged?
>> Thanks for any advices/hints
>> Clemens
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


AW: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Clemens Wyss DEV <cl...@mysign.ch>.
Sorry for being imprecise
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

- Clemens
-----Ursprüngliche Nachricht-----
Von: Michael McCandless [mailto:lucene@mikemccandless.com] 
Gesendet: Sonntag, 18. Mai 2014 16:51
An: Lucene Users
Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

But what is the output of "java -fullversion"?

Mike McCandless

http://blog.mikemccandless.com

On Sun, May 18, 2014 at 5:24 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>> What java version?
> We are still on (1.)6
>
> -----Ursprüngliche Nachricht-----
> Von: Michael McCandless [mailto:lucene@mikemccandless.com]
> Gesendet: Donnerstag, 15. Mai 2014 11:43
> An: Lucene Users
> Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>
> delGen=-1 means there are no deletions, but the exception makes no 
> sense because up above SegmentReader.java calls si.hasDeletions() 
> which returns delGen != -1 which should have mean 
> Lucene40LiveDocsFormat.readLiveDocs should not have been called.  It 
> seems impossible :)
>
> What java version?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Wed, May 14, 2014 at 2:16 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>> Tackled this down a little bit more:
>> Lucene40LiveDocsFormat#readLiveDocs calls 
>> IndexFileNames#fileNameForGeneration
>> If I get this right, param 'gen' seems to be -1.
>> Gen is being gathered from SegmentCommitInfo#getDelGen
>>
>> What does "delGen == -1" mean?
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Clemens Wyss DEV [mailto:clemensdev@mysign.ch]
>> Gesendet: Dienstag, 13. Mai 2014 18:23
>> An: java-user@lucene.apache.org
>> Betreff: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>>
>> I am facing the following stacktrace:
>>         java.lang.NullPointerException: null
>>           at java.io.File.<init>(File.java:305) ~[na:1.6.0_26]
>>           at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.codecs.lucene40.BitVector.<init>(BitVector.java:327) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:90) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:131) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:194) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:843) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:250) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at
>> org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.
>> java:170) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 
>> 11:05:50]
>>
>> Unfortunately not always. Looks like j.i.File<init> is getting param child == null What might go wrong in openIfChanged?
>> Thanks for any advices/hints
>> Clemens
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Michael McCandless <lu...@mikemccandless.com>.
But what is the output of "java -fullversion"?

Mike McCandless

http://blog.mikemccandless.com

On Sun, May 18, 2014 at 5:24 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>> What java version?
> We are still on (1.)6
>
> -----Ursprüngliche Nachricht-----
> Von: Michael McCandless [mailto:lucene@mikemccandless.com]
> Gesendet: Donnerstag, 15. Mai 2014 11:43
> An: Lucene Users
> Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>
> delGen=-1 means there are no deletions, but the exception makes no sense because up above SegmentReader.java calls si.hasDeletions() which returns delGen != -1 which should have mean Lucene40LiveDocsFormat.readLiveDocs should not have been called.  It seems impossible :)
>
> What java version?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Wed, May 14, 2014 at 2:16 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
>> Tackled this down a little bit more:
>> Lucene40LiveDocsFormat#readLiveDocs calls
>> IndexFileNames#fileNameForGeneration
>> If I get this right, param 'gen' seems to be -1.
>> Gen is being gathered from SegmentCommitInfo#getDelGen
>>
>> What does "delGen == -1" mean?
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Clemens Wyss DEV [mailto:clemensdev@mysign.ch]
>> Gesendet: Dienstag, 13. Mai 2014 18:23
>> An: java-user@lucene.apache.org
>> Betreff: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>>
>> I am facing the following stacktrace:
>>         java.lang.NullPointerException: null
>>           at java.io.File.<init>(File.java:305) ~[na:1.6.0_26]
>>           at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.codecs.lucene40.BitVector.<init>(BitVector.java:327) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:90) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:131) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:194) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:843) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:250) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>>           at
>> org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.
>> java:170) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19
>> 11:05:50]
>>
>> Unfortunately not always. Looks like j.i.File<init> is getting param child == null What might go wrong in openIfChanged?
>> Thanks for any advices/hints
>> Clemens
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


AW: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Clemens Wyss DEV <cl...@mysign.ch>.
> What java version?
We are still on (1.)6

-----Ursprüngliche Nachricht-----
Von: Michael McCandless [mailto:lucene@mikemccandless.com] 
Gesendet: Donnerstag, 15. Mai 2014 11:43
An: Lucene Users
Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

delGen=-1 means there are no deletions, but the exception makes no sense because up above SegmentReader.java calls si.hasDeletions() which returns delGen != -1 which should have mean Lucene40LiveDocsFormat.readLiveDocs should not have been called.  It seems impossible :)

What java version?

Mike McCandless

http://blog.mikemccandless.com


On Wed, May 14, 2014 at 2:16 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
> Tackled this down a little bit more:
> Lucene40LiveDocsFormat#readLiveDocs calls 
> IndexFileNames#fileNameForGeneration
> If I get this right, param 'gen' seems to be -1.
> Gen is being gathered from SegmentCommitInfo#getDelGen
>
> What does "delGen == -1" mean?
>
> -----Ursprüngliche Nachricht-----
> Von: Clemens Wyss DEV [mailto:clemensdev@mysign.ch]
> Gesendet: Dienstag, 13. Mai 2014 18:23
> An: java-user@lucene.apache.org
> Betreff: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>
> I am facing the following stacktrace:
>         java.lang.NullPointerException: null
>           at java.io.File.<init>(File.java:305) ~[na:1.6.0_26]
>           at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.codecs.lucene40.BitVector.<init>(BitVector.java:327) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:90) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:131) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:194) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:843) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:250) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at 
> org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.
> java:170) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 
> 11:05:50]
>
> Unfortunately not always. Looks like j.i.File<init> is getting param child == null What might go wrong in openIfChanged?
> Thanks for any advices/hints
> Clemens
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Michael McCandless <lu...@mikemccandless.com>.
delGen=-1 means there are no deletions, but the exception makes no
sense because up above SegmentReader.java calls si.hasDeletions()
which returns delGen != -1 which should have mean
Lucene40LiveDocsFormat.readLiveDocs should not have been called.  It
seems impossible :)

What java version?

Mike McCandless

http://blog.mikemccandless.com


On Wed, May 14, 2014 at 2:16 AM, Clemens Wyss DEV <cl...@mysign.ch> wrote:
> Tackled this down a little bit more:
> Lucene40LiveDocsFormat#readLiveDocs calls IndexFileNames#fileNameForGeneration
> If I get this right, param 'gen' seems to be -1.
> Gen is being gathered from SegmentCommitInfo#getDelGen
>
> What does "delGen == -1" mean?
>
> -----Ursprüngliche Nachricht-----
> Von: Clemens Wyss DEV [mailto:clemensdev@mysign.ch]
> Gesendet: Dienstag, 13. Mai 2014 18:23
> An: java-user@lucene.apache.org
> Betreff: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>
> I am facing the following stacktrace:
>         java.lang.NullPointerException: null
>           at java.io.File.<init>(File.java:305) ~[na:1.6.0_26]
>           at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.codecs.lucene40.BitVector.<init>(BitVector.java:327) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:90) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:131) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:194) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:843) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:250) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>           at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
>
> Unfortunately not always. Looks like j.i.File<init> is getting param child == null What might go wrong in openIfChanged?
> Thanks for any advices/hints
> Clemens
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


AW: [lucene 4.6] NPE when calling IndexReader#openIfChanged

Posted by Clemens Wyss DEV <cl...@mysign.ch>.
Tackled this down a little bit more:
Lucene40LiveDocsFormat#readLiveDocs calls IndexFileNames#fileNameForGeneration
If I get this right, param 'gen' seems to be -1.
Gen is being gathered from SegmentCommitInfo#getDelGen

What does "delGen == -1" mean?

-----Ursprüngliche Nachricht-----
Von: Clemens Wyss DEV [mailto:clemensdev@mysign.ch] 
Gesendet: Dienstag, 13. Mai 2014 18:23
An: java-user@lucene.apache.org
Betreff: [lucene 4.6] NPE when calling IndexReader#openIfChanged

I am facing the following stacktrace:
	java.lang.NullPointerException: null
	  at java.io.File.<init>(File.java:305) ~[na:1.6.0_26]
	  at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.codecs.lucene40.BitVector.<init>(BitVector.java:327) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:90) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:131) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:194) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:843) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:250) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]
	  at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170) ~[lucene-core.jar:4.6.0 1543363 - simon - 2013-11-19 11:05:50]

Unfortunately not always. Looks like j.i.File<init> is getting param child == null What might go wrong in openIfChanged?
Thanks for any advices/hints
Clemens

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org