You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "david-sitsky (via GitHub)" <gi...@apache.org> on 2023/03/06 05:06:19 UTC
[GitHub] [lucene] david-sitsky opened a new issue, #12185: Using DirectIODirectory results in BufferOverflowException
david-sitsky opened a new issue, #12185:
URL: https://github.com/apache/lucene/issues/12185
### Description
I have a large Lucene-based application that I have recently upgraded to 9.5.0. All works fine with the existing Directory implementations, but when I try using DirectIODirectory, I get errors like the following:
```
java.nio.BufferOverflowException: null
at java.nio.Buffer.nextPutIndex(Buffer.java:666) ~[?:?]
at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:331) ~[?:?]
at org.apache.lucene.misc.store.DirectIODirectory$DirectIOIndexOutput.writeByte(DirectIODirectory.java:232) ~[lucene-misc-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 - 2023-01-25 16:44:59]
at org.apache.lucene.codecs.CodecUtil.writeBEInt(CodecUtil.java:653) ~[lucene-core-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 - 2023-01-25 16:44:59]
at org.apache.lucene.codecs.CodecUtil.writeHeader(CodecUtil.java:82) ~[lucene-core-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 - 2023-01-25 16:44:59]
at org.apache.lucene.codecs.CodecUtil.writeIndexHeader(CodecUtil.java:125) ~[lucene-core-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 - 2023-01-25 16:44:59]
at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.<init>(Lucene90CompressingStoredFieldsWriter.java:137) ~[lucene-core-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 - 2023-0
1-25 16:44:59]
at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsFormat.fieldsWriter(Lucene90CompressingStoredFieldsFormat.java:140) ~[lucene-core-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 -
2023-01-25 16:44:59]
at org.apache.lucene.codecs.lucene90.Lucene90StoredFieldsFormat.fieldsWriter(Lucene90StoredFieldsFormat.java:154) ~[lucene-core-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 - 2023-01-25 16:44:59]
at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:233) ~[lucene-core-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 - 2023-01-25 16:44:59]
at org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:273) ~[lucene-core-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 - 2023-01-25 16:44:59]
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:110) ~[lucene-core-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 - 2023-01-25 16:44:59]
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5140) ~[lucene-core-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 - 2023-01-25 16:44:59]
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4680) ~[lucene-core-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 - 2023-01-25 16:44:59]
at org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6432) ~[lucene-core-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 - 2023-01-25 16:44:59]
at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:38) ~[lucene-core-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 - 2023-01-25 16:44:59]
at com.nuix.load.lucene.s$a.merge(SourceFile:330) ~[nuix-processing-9.11.2.3097+sits-7cf07c727f8.jar:9.11.2.3097+sits-7cf07c727f8]
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3693) ~[lucene-core-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 - 2023-01-25 16:44:59]
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:4043) ~[lucene-core-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 - 2023-01-25 16:44:59]
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4005) ~[lucene-core-9.5.0.jar:9.5.0 13803aa6ea7fee91f798cfeded4296182ac43a21 - 2023-01-25 16:44:59]
```
### Version and environment details
9.5.0 on Linux using DirectIODirectory with NIOFSDirectory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [GitHub] [lucene] david-sitsky commented on issue #12185: Using DirectIODirectory results in BufferOverflowException
Posted by Michael Sokolov <ms...@gmail.com>.
Using directio with nfs makes no sense at all to me, I think that is the
problem in a nutshell. Directio tries to bypass the operating systems
buffers, but that's not going to play nicely with nfs.
On Wed, Mar 22, 2023, 4:38 PM david-sitsky (via GitHub) <gi...@apache.org>
wrote:
>
> david-sitsky commented on issue #12185:
> URL: https://github.com/apache/lucene/issues/12185#issuecomment-1480390389
>
> As an aside, in some standard benchmark tests I run with our product, I
> have found the final optimisation of Lucene indexes after all the data has
> been indexed took 36 seconds with NIO, but 148 seconds with NIO+DirectIO
> enabled. For mmap, optimisation took 30 seconds but 100 seconds with
> DirectIO was enabled. So it is odd the use-case DirectIO was meant to
> speed up actually seemed to be slower..
>
>
> --
> This is an automated message from the Apache Git Service.
> To respond to the message, please log on to GitHub and use the
> URL above to go to the specific comment.
>
> To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
>
> For queries about this service, please contact Infrastructure at:
> users@infra.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
> For additional commands, e-mail: issues-help@lucene.apache.org
>
>
[GitHub] [lucene] mikemccand commented on issue #12185: Using DirectIODirectory results in BufferOverflowException
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #12185:
URL: https://github.com/apache/lucene/issues/12185#issuecomment-1469863743
Hmm, I think the root cause exception is essentially [thrown from here](https://github.com/openjdk/jdk17/blob/master/src/java.base/share/classes/java/nio/Buffer.java#L722), which means it looks like `DirectIOIndexOutput` failed to flush its direct `ByteBuffer` when it was full. Not sure how that could happen on quickly staring at the code.
Does this consistently reproduce? Looks like you are using `SerialMergeScheduler` so maybe it does (no thread scheduling "randomness")?
Which JDK are you running? And which Linux flavor/version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] MarcusSorealheis commented on issue #12185: Using DirectIODirectory results in BufferOverflowException
Posted by "MarcusSorealheis (via GitHub)" <gi...@apache.org>.
MarcusSorealheis commented on issue #12185:
URL: https://github.com/apache/lucene/issues/12185#issuecomment-1475469270
For folks following along,
https://github.com/apache/lucene/blob/0782535017c9e737350e96fb0f53457c7b8ecf03/lucene/MIGRATE.md?plain=1#L171-L176
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mikemccand commented on issue #12185: Using DirectIODirectory results in BufferOverflowException
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #12185:
URL: https://github.com/apache/lucene/issues/12185#issuecomment-1481243567
> Using the standard Directory implementations of Lucene work fine on this NFS filesystem, so perhaps there is a corner case here which is not handled?
Ahh this is also interesting. NFS is tricky (buggy/laggy caching, no "delete on final close" semantics, stale files, etc.), but I'm not yet sure why in particular it would run amock of `DirectIO`...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mikemccand commented on issue #12185: Using DirectIODirectory results in BufferOverflowException
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #12185:
URL: https://github.com/apache/lucene/issues/12185#issuecomment-1481241581
> As an aside, in some standard benchmark tests I run with our product, I have found the final optimisation of Lucene indexes after all the data has been indexed took 36 seconds with NIO, but 148 seconds with NIO+DirectIO enabled. For mmap, optimisation took 30 seconds but 100 seconds with DirectIO was enabled. So it is odd the use-case DirectIO was meant to speed up actually seemed to be slower..
That's a great datapoint -- thanks for sharing!
Maybe try increasing the `mergeBufferSize` `DirectIO` is using? And maybe also the `minBytesDirect`?
We should not expect `DirectIO` to be faster: it is bypassing the OS's (helpful!) write buffer cache and taking caching into its own hands, apparently quite a bit less effectively. What it should be helpful for is applications that are also doing concurrent searching on the same hardware. In this case the `DirectIO` avoids conflicting with the hot cached pages for searching, hopefully reducing long-pole latencies of queries impacted by merging. But a 3-4X slowdown on merging is horrible :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] david-sitsky commented on issue #12185: Using DirectIODirectory results in BufferOverflowException
Posted by "david-sitsky (via GitHub)" <gi...@apache.org>.
david-sitsky commented on issue #12185:
URL: https://github.com/apache/lucene/issues/12185#issuecomment-1478677192
Hi Mike - this is on Ubuntu 18.04.5 LTS, kernel 4.15.0-151-generic, OpenJDK 64-Bit Server VM Temurin-11.0.16.1+1.
However I have noticed this fails seemingly only when I am using an NFS filesystem, as opposed to a local filesystem. I realise using a local filesystem should be preferred in all cases, but sometimes a network filesystem is required for some of our use-cases.
Using the standard Directory implementations of Lucene work fine on this NFS filesystem, so perhaps there is a corner case here which is not handled?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] david-sitsky commented on issue #12185: Using DirectIODirectory results in BufferOverflowException
Posted by "david-sitsky (via GitHub)" <gi...@apache.org>.
david-sitsky commented on issue #12185:
URL: https://github.com/apache/lucene/issues/12185#issuecomment-1480390389
As an aside, in some standard benchmark tests I run with our product, I have found the final optimisation of Lucene indexes after all the data has been indexed took 36 seconds with NIO, but 148 seconds with NIO+DirectIO enabled. For mmap, optimisation took 30 seconds but 100 seconds with DirectIO was enabled. So it is odd the use-case DirectIO was meant to speed up actually seemed to be slower..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org