You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "itschrispeck (via GitHub)" <gi...@apache.org> on 2023/08/22 00:38:05 UTC
[GitHub] [pinot] itschrispeck opened a new issue, #11402: Mutable Lucene Index 'breaks' after heavy query
itschrispeck opened a new issue, #11402:
URL: https://github.com/apache/pinot/issues/11402
#### Symptoms:
After a heavy query, no more values can be inserted into the mutable Lucene index for a segment.
#### Reproduction:
Lucene index a large amount of data, then execute a `text_match` filter that will time out or a query with a number of text_matches such that at least one will time out. Following this `failed to index value with text_index` will be logged for all values until the segment is sealed.
#### Initial investigation:
It appears that this a byproduct of a limitation of FSDirectories that support concurrency ([NIOFSDirectory](https://lucene.apache.org/core/8_2_0/core/org/apache/lucene/store/NIOFSDirectory.html)/[MMapDirectory](https://lucene.apache.org/core/8_2_0/core/org/apache/lucene/store/MMapDirectory.html)):
>NOTE: Accessing this class either directly or indirectly from a thread while it's interrupted can close the underlying file descriptor immediately if at the same time the thread is blocked on IO. The file descriptor will remain closed and subsequent access to [NIOFSDirectory](https://lucene.apache.org/core/8_2_0/core/org/apache/lucene/store/NIOFSDirectory.html) will throw a [ClosedChannelException](https://docs.oracle.com/javase/8/docs/api/java/nio/channels/ClosedChannelException.html?is-external=true). If your application uses either [Thread.interrupt()](https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.html?is-external=true#interrupt--) or [Future.cancel(boolean)](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Future.html?is-external=true#cancel-boolean-) you should use the legacy RAFDirectory from the Lucene misc module in favor of [NIOFSDirectory](https://lucene.apache.org/core/8_2_0/core/org/apache/lucene/store/NIOFSDirectory.html).
Presumably a slow query gets canceled/timed out while thread is blocked on IO. The channel is closed preventing future events from occurring normally.
I think this should affect queries against the consuming segment as well, but have not yet verified it.
ClosedChannelException:
```
j.n.c.ClosedChannelException: null at sun.nio.ch.FileLockImpl.release(FileLockImpl.java:58)
at java.nio.channels.FileLock.close(FileLock.java:317)
at o.a.l.s.NativeFSLockFactory$NativeFSLock.close(NativeFSLockFactory.java:206)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:89) at org.apache.lucene.util.IOUtils.close(IOUtils.java:77)
at o.a.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2337)
at o.a.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2268)
at o.a.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:4910)
at o.a.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1610)
at o.a.lucene.index.IndexWriter.addDocument(IndexWriter.java:1213)
at o.a.p.s.l.s.c.i.t.LuceneTextIndexCreator.add(LuceneTextIndexCreator.java:136) <...>
```
Insert failure:
```
java.lang.RuntimeException: Caught exception while adding a new document to the Lucene index for column: col_name
at org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator.add(LuceneTextIndexCreator.java:138)
at org.apache.pinot.segment.local.realtime.impl.invertedindex.RealtimeLuceneTextIndex.add(RealtimeLuceneTextIndex.java:93)
at org.apache.pinot.segment.spi.index.mutable.MutableTextIndex.add(MutableTextIndex.java:29)
at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.addNewRow(MutableSegmentImpl.java:694)
at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:502)
<...>
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:681)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:695)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1591)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1213)
at org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator.add(LuceneTextIndexCreator.java:136)
... 8 common frames omitted
```
Refresh failure:
```
Caused by: org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/<...>/dataDir/table_name_REALTIME/consumers/table_name__11__15__20230811T1226Z/col_name.lucene.index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2023-08-11T12:26:13.731708Z)
at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:178)
at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
at org.apache.lucene.codecs.lucene80.Lucene80NormsConsumer.<init>(Lucene80NormsConsumer.java:45)
at org.apache.lucene.codecs.lucene80.Lucene80NormsFormat.normsConsumer(Lucene80NormsFormat.java:85)
at org.apache.lucene.index.DefaultIndexingChain.writeNorms(DefaultIndexingChain.java:304)
at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:131)
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:468)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:555)
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:722)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:494)
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:297)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:272)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262)
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:165)
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156)
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225)
at org.apache.pinot.segment.local.realtime.impl.invertedindex.RealtimeLuceneIndexReaderRefreshThread.run(RealtimeLuceneIndexReaderRefreshThread.java:126)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Mutable Lucene Index 'breaks' after heavy query [pinot]
Posted by "itschrispeck (via GitHub)" <gi...@apache.org>.
itschrispeck closed issue #11402: Mutable Lucene Index 'breaks' after heavy query
URL: https://github.com/apache/pinot/issues/11402
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org