[jira] [Commented] (LUCENE-6169) Recent Java 9 commit breaks fsync on directory

["Uwe Schindler (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14549288#comment-14549288 ] 

Uwe Schindler commented on LUCENE-6169:
---------------------------------------

The commit in the OpenJDK 9 tree is about to be removed:

{quote}
We have revised our approach to this (pair or trio of) problem(s). The following sequence of actions is proposed.

1. Revert the patch which fixed https://bugs.openjdk.java.net/browse/JDK-8066915, (fs) Files.newByteChannel opens directories for cases where subsequent reads may fail, which instigated the present situation. I will file and post the link to a new Issue for this.

2. Work on a fix for https://bugs.openjdk.java.net/browse/JDK-8080589, (fc) FileChannel.force should use fcntl(F_FULLFSYNC) instead of fsync on OS X.

3. Work on a zero to minimal impact fix for https://bugs.openjdk.java.net/browse/JDK-8080235, (fs) Provide ability to flush all modified buffered data to a permanent storage device.

Hopefully the foregoing plan of record shall be to everyone’s satisfaction and will converge to an eventual mutually acceptable, logical solution.

Thanks,

Brian
{quote}

> Recent Java 9 commit breaks fsync on directory
> ----------------------------------------------
>
>                 Key: LUCENE-6169
>                 URL: https://issues.apache.org/jira/browse/LUCENE-6169
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/store
>            Reporter: Uwe Schindler
>              Labels: Java9
>
> I open this issue to keep track of the communication with Oracle and OpenJDK about this:
> Basically, what happens: In LUCENE-5588 we added support to FSDirectory to be able to sync on directory metadata changes (means the contents of the directory itsself). This is very important on Unix system (maybe also on Windows), because fsyncing a single file does not necessarily writes the directory's contents to disk. Lucene uses this for commits. We first do an atomic rename of the segments file  (to make the commit public), but we have to be sure that the rename operation is written to disk. Because of that we must fsync the directory.
> To enforce this with plain system calls (libc), you open a directory for read and then call fsync. In java this can be done by opening a FileChannel on the direczory(for read) and call fc.force() on it.
> Unfortunately the commit http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/e5b66323ae45 in OpenJDK 9 break this. The corresponding issue is https://bugs.openjdk.java.net/browse/JDK-8066915. The JDK now explicitly checks if a file is a directory and disallows opening a FileChannel on it. This breaks our commit safety.
> Because this behaviour is undocumented (not even POSIX has explicit semantics for syncing directories), we know that it worked at least on MacOSX and Linux. The code in IOUtils is currently written in a way that it tries to sync the diretory, but swallows any Exception. So this change does not break Liucene, but it breaks our commit safety. During testing we assert that the fsync actually works on Linux and MacOSX, in production code the user will notice nothing.
> We should take action and contact Alan Bateman about his commit and this issue on the mailing list, possibly through Rory O'Donnell.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org