You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Simon Willnauer (Jira)" <ji...@apache.org> on 2020/08/17 09:50:00 UTC
[jira] [Commented] (LUCENE-9467) TestOfflineSorter fails with
combination of NRTCaching and ByteBuffersDirectory
[ https://issues.apache.org/jira/browse/LUCENE-9467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17178876#comment-17178876 ]
Simon Willnauer commented on LUCENE-9467:
-----------------------------------------
one option to fix this is to use fileLength instead of opening a file.
{noformat}
diff --git a/lucene/core/src/java/org/apache/lucene/store/NRTCachingDirectory.java b/lucene/core/src/java/org/apache/lucene/store/NRTCachingDirectory.java
index 597ea9d5f2e..40e798959c1 100644
--- a/lucene/core/src/java/org/apache/lucene/store/NRTCachingDirectory.java
+++ b/lucene/core/src/java/org/apache/lucene/store/NRTCachingDirectory.java
@@ -277,7 +277,7 @@ public class NRTCachingDirectory extends FilterDirectory implements Accountable
* there's some unexpected error. */
static boolean slowFileExists(Directory dir, String fileName) throws IOException {
try {
- dir.openInput(fileName, IOContext.DEFAULT).close();
+ dir.fileLength(fileName);
return true;
} catch (NoSuchFileException | FileNotFoundException e) {
return false;
{noformat}
> TestOfflineSorter fails with combination of NRTCaching and ByteBuffersDirectory
> -------------------------------------------------------------------------------
>
> Key: LUCENE-9467
> URL: https://issues.apache.org/jira/browse/LUCENE-9467
> Project: Lucene - Core
> Issue Type: Bug
> Affects Versions: 8.7, 8.6.1
> Reporter: Simon Willnauer
> Priority: Major
>
> TestOfflineSorter#testThreadSafety fails on 8.x since NRTCachingDirectory uses the delegates openInput method to slowly check if the file exists. This is not present anymore on main line. The problem is that ByteBuffersDirectory fires an AccessDeniedException when the file is not fully written yet.
> {noformat}
> 00:27:31 [junit4] Suite: org.apache.lucene.util.TestOfflineSorter
> 00:27:31 [junit4] 2> KMN 14, 2020 6:27:30 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
> 00:27:31 [junit4] 2> WARNING: Uncaught exception in thread: Thread[Thread-7318,5,TGRP-TestOfflineSorter]
> 00:27:31 [junit4] 2> java.lang.RuntimeException: java.nio.file.AccessDeniedException: Can't open a file still open for writing: unsorted_tmp_6.tmp
> 00:27:31 [junit4] 2> at __randomizedtesting.SeedInfo.seed([B1129484013118F7]:0)
> 00:27:31 [junit4] 2> at org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:272)
> 00:27:31 [junit4] 2> Caused by: java.nio.file.AccessDeniedException: Can't open a file still open for writing: unsorted_tmp_6.tmp
> 00:27:31 [junit4] 2> at org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:251)
> 00:27:31 [junit4] 2> at org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:218)
> 00:27:31 [junit4] 2> at org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:233)
> 00:27:31 [junit4] 2> at org.apache.lucene.store.NRTCachingDirectory.slowFileExists(NRTCachingDirectory.java:280)
> 00:27:31 [junit4] 2> at org.apache.lucene.store.NRTCachingDirectory.createTempOutput(NRTCachingDirectory.java:254)
> 00:27:31 [junit4] 2> at org.apache.lucene.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:707)
> 00:27:31 [junit4] 2> at org.apache.lucene.util.TestOfflineSorter.checkSort(TestOfflineSorter.java:184)
> 00:27:31 [junit4] 2> at org.apache.lucene.util.TestOfflineSorter.access$100(TestOfflineSorter.java:48)
> 00:27:31 [junit4] 2> at org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:268)
> 00:27:31 [junit4] 2>
> 00:27:31 [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestOfflineSorter -Dtests.method=testThreadSafety -Dtests.seed=B1129484013118F7 -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=cgg-UG -Dtests.timezone=Asia/Dacca -Dtests.asserts=true -Dtests.file.encoding=UTF8
> 00:27:31 [junit4] ERROR 0.04s J3 | TestOfflineSorter.testThreadSafety <<<
> 00:27:31 [junit4] > Throwable #1: java.lang.AssertionError
> 00:27:31 [junit4] > at org.apache.lucene.util.TestOfflineSorter.testThreadSafety(TestOfflineSorter.java:283)
> 00:27:31 [junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 00:27:31 [junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 00:27:31 [junit4] > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 00:27:31 [junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 00:27:31 [junit4] > at java.base/java.lang.Thread.run(Thread.java:834)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7993, name=Thread-7318, state=RUNNABLE, group=TGRP-TestOfflineSorter]
> 00:27:31 [junit4] > Caused by: java.lang.RuntimeException: java.nio.file.AccessDeniedException: Can't open a file still open for writing: unsorted_tmp_6.tmp
> 00:27:31 [junit4] > at __randomizedtesting.SeedInfo.seed([B1129484013118F7]:0)
> 00:27:31 [junit4] > at org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:272)
> 00:27:31 [junit4] > Caused by: java.nio.file.AccessDeniedException: Can't open a file still open for writing: unsorted_tmp_6.tmp
> 00:27:31 [junit4] > at org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:251)
> 00:27:31 [junit4] > at org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:218)
> 00:27:31 [junit4] > at org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:233)
> 00:27:31 [junit4] > at org.apache.lucene.store.NRTCachingDirectory.slowFileExists(NRTCachingDirectory.java:280)
> 00:27:31 [junit4] > at org.apache.lucene.store.NRTCachingDirectory.createTempOutput(NRTCachingDirectory.java:254)
> 00:27:31 [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:707)
> 00:27:31 [junit4] > at org.apache.lucene.util.TestOfflineSorter.checkSort(TestOfflineSorter.java:184)
> 00:27:31 [junit4] > at org.apache.lucene.util.TestOfflineSorter.access$100(TestOfflineSorter.java:48)
> 00:27:31 [junit4] > at org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:268)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org