You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2010/12/03 02:08:11 UTC

[jira] Created: (LUCENE-2791) WindowsDirectory

WindowsDirectory
----------------

                 Key: LUCENE-2791
                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
             Project: Lucene - Java
          Issue Type: New Feature
          Components: Store
            Reporter: Robert Muir
         Attachments: LUCENE-2791.patch

We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966496#action_12966496 ] 

Robert Muir commented on LUCENE-2791:
-------------------------------------

{quote}
To my above comment and OVERLAPPED in my opinion not realy used (as not async):
Reading http://msdn.microsoft.com/en-us/library/aa365467(VS.85).aspx#synchronization_and_file_position it seems to behave little different when OVERLAPPED is non-null for synchronous reads (as we do currently). But as you never read the contents og OVERLAPPED after the function call, why pass it in? Please explain 
{quote}

To position the read versus seeking the file. This is explained in the documentation on that very same page!:

Considerations for working with *synchronous* file handles:

If lpOverlapped is not NULL, the read operation *starts at the offset that is specified* in the OVERLAPPED structure and ReadFile does not return until the read operation is complete. The system updates the OVERLAPPED offset before ReadFile returns.


> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Issue Comment Edited: (LUCENE-2791) WindowsDirectory

Posted by "Native Policeman (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966486#action_12966486 ] 

Native Policeman edited comment on LUCENE-2791 at 12/3/10 6:11 AM:
-------------------------------------------------------------------

There are some violations:

bq. We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

Ahm you dont really use overlapped/async IO, ReadFile in your code only returns when read operation is finished, so not async. Because when you do you have to pass a flag: FILE_FLAG_OVERLAPPED. But I don't think we should really make it async for now, that is too much stuff as Lucene does not really support it at the moment.

- Currently you don't really need the OVERLAPPED structure at all, just pass NULL: [http://msdn.microsoft.com/en-us/library/aa365467(VS.85).aspx], or is there a reason?
- Do we need FILE_SHARE_WRITE?
- You use env->GetStringUTFChars, which is not really a problem as index filenames are always ASCII only. The default methods in the Win32 API use the system default charset (ANSI) for the arguments to OpenFile, which links to "OpenFileA" in kernel32.dll. I would suggest to use the real UTF16 chars (env->GetStringChars) and use OpenFileW method, which takes wchar instead of char. The same applies to Linux Dir, but there I am not sure how encodings are handled.
- Also the Format functions for GetLastError() results use ANSI per default. We should use the *W method, too, and pass it to java as wchar.
- Exception handling: in native read, you call throwIOException, when ReadFile() returns false. But 2 lines later you still call the method to copy the local buffer to the java byte array. JNI docs say, that after an Exception is in the thread's local exception, you should not call any JNI methods anymore (expect some allowed ones). Also the copy operation is not really needed.
- As in contract to the native Linux directory we are using our own Handles and not Java's FileDescriptor class to store the handle. If something goes wrong, GC can never clean up the file and it stays open. So you should add a finalizer, Java's internal File methods do that also (RandomAccessFile has a finalizer that calls close on operating system file).

      was (Author: nativepol):
    There are some violations:

bq. We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

Ahm you dont really use overlapped/async IO, ReadFile in your code only returns when read operation is finished, so not async. Because when you do you have to pass a flag: FILE_FLAG_OVERLAPPED. But I don't think we should really make it async for now, that is too much stuff as Lucene does not really support it at the moment.

- Currently you don't really need the OVERLAPPED structure at all, just pass NULL: [http://msdn.microsoft.com/en-us/library/aa365467(VS.85).aspx], or is there a reason?
- Do we need FILE_SHARE_WRITE?
- You use env->GetStringUTFChars, which is not really a problem as index filenames are always ASCII only. The default methods in the Win32 API use the system default charset (ANSI) for the arguments to OpenFile, which liks to "OpenFileA" in kernel32.dll. I would suggest to use the reaö UTF16 chars (env->GetStringChars) and use OpenFileW method, which takes wchar instead of char. The same applies to Linux Dir, but there I am not sure how encodings are handled.
- Exception handling: in native read, you call throwIOException, when ReadFile() returns false. But 2 lines later you still call the method to copy the local buffer to the java byte array. JNI docs say, that after an Exception is in the thread's local exception, you should not call any JNI methods anymore (expect some allowed ones). Also the copy operation is not really needed.
- As in contract to the native Linux directory we are using our own Handles and not Java's FileDescriptor class to store the handle. If something goes wrong, GC can never clean up the file and it stays open. So you should add a finalizer, Java's internal File methods do that also (RandomAccessFile has a finalizer that calls close on operating system file).
  
> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Native Policeman (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966497#action_12966497 ] 

Native Policeman commented on LUCENE-2791:
------------------------------------------

bq. So bottom line: I disagree with you on everything you said.

This was not against you, but I am hacking native Windows since Win32 was started :-)

bq. Yes I do use it, do you understand C ?

I kill you the next time we meet. I am hacking C longer than Java. This was not about C it was about the MSDN docs telling something different. See my last comment above. Thats all. Why are you so aggresive?

{quote}
bq. Exception handling: in native read, you call throwIOException, when ReadFile() returns false. But 2 lines later you still call the method to copy the local buffer to the java byte array. JNI docs say, that after an Exception is in the thread's local exception, you should not call any JNI methods anymore (expect some allowed ones). Also the copy operation is not really needed.

I don't think you know what you are talking about!
{quote}

Sorry we seem to have a misunderstanding. I was *not* talking about the internal copy the JVM does. I was talking about the fact, that you should not call JNI functions anymore when you set the thread's exception status and exit the function as soon as possible. You set this status after the failed ReadFile with throwIOEx(). After that its no longer needed to copy the buffer[] to the java byte array and its risky to do it because I am not sure if this env->SetByteArrayRegion() is safe to be called after exception status is set - that all I wanted to say. I would change tha code to:

{code}
if (ReadFile((HANDLE) fd, &buffer, length, &numRead, &io)) {
  env->SetByteArrayRegion(bytes, offset, numRead, (const jbyte *) buffer);
} else {
  throwIOException(env, GetLastError());
  numRead = -1;
}  	
{code}

Please note env->throwNew() always returns, it simply sets the thread's exception status, so the JVM thorws the Java Exceüption passed in after the native method returns to Java bytecode again.

Thanks,
Native Policeman

> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Resolved: (LUCENE-2791) WindowsDirectory

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Muir resolved LUCENE-2791.
---------------------------------

       Resolution: Fixed
    Fix Version/s: 4.0
                   3.1
         Assignee: Robert Muir

committed revision 1041844 (trunk), 1041881 (3x).


> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>            Assignee: Robert Muir
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Native Policeman (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966523#action_12966523 ] 

Native Policeman commented on LUCENE-2791:
------------------------------------------

*wink*

> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966522#action_12966522 ] 

Uwe Schindler commented on LUCENE-2791:
---------------------------------------

The Native Policeman now retires!

> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966459#action_12966459 ] 

Michael McCandless commented on LUCENE-2791:
--------------------------------------------

This is awesome!  It gives us a workaround to the JRE bug (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734) killing NIOFSDir's concurrency on Windows.

Does is also allow for deletion of still-open files?

You may want to experiment w/ different buffer sizes to see if that impacts performance.

Also: this allows you to do the equivalent of Linux's O_DIRECT, right?  If so... I think we should augment the Dir API so that on opening an input it receives some sort of IOContext or something expressing why we are opening.  So for merging we would want "direct iO" (bypass OS buffer cache or NOREUSE fadvise flag, etc.), and larger buffer sizes and perhaps other low level IO settings in the future.

> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Updated: (LUCENE-2791) WindowsDirectory

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Muir updated LUCENE-2791:
--------------------------------

    Attachment: WindowsDirectory.dll

heres a dll built for 32-bit windows.
to use it, just put it in a folder thats in your windows PATH, then the directory will work.

to then run all lucene tests with it, first compile contrib/misc, then run a command like this:

ant -lib \path\to\lusolr\lucene\build\contrib\misc\lucene-misc-4.0-SNAPSHOT.jar test-core -Dtests.directory=WindowsDirectory


> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, WindowsDirectory.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966501#action_12966501 ] 

Robert Muir commented on LUCENE-2791:
-------------------------------------

bq. This was not against you, but I am hacking native Windows since Win32 was started 

But you don't understand how its IO works?

> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "DM Smith (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966540#action_12966540 ] 

DM Smith commented on LUCENE-2791:
----------------------------------

I've just back ported all the code to Java 1.1. Also, this port also deletes everything but 7-bit ASCII.

(Just couldn't resist....)



> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Native Policeman (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966493#action_12966493 ] 

Native Policeman commented on LUCENE-2791:
------------------------------------------

To my above comment and OVERLAPPED in my opinion not realy used (as not async):
Reading [http://msdn.microsoft.com/en-us/library/aa365467(VS.85).aspx#synchronization_and_file_position] it seems to behave little different when OVERLAPPED is non-null for synchronous reads (as we do currently). But as you never read the contents og OVERLAPPED after the function call, why pass it in? Please explain :-)

> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966524#action_12966524 ] 

Uwe Schindler commented on LUCENE-2791:
---------------------------------------

I was thinking maybe I retire as generics policeman too.

I decided we should convert all of our code back to java 1.4, generics are useless and should be abolished!

> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Native Policeman (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966503#action_12966503 ] 

Native Policeman commented on LUCENE-2791:
------------------------------------------

bq. If lpOverlapped is not NULL, the read operation starts at the offset that is specified in the OVERLAPPED structure and ReadFile does not return until the read operation is complete. The system updates the OVERLAPPED offset before ReadFile returns.

Thanks, I missed that. Sorry - the idea is simply to make positioned reads, I understand that.

> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966521#action_12966521 ] 

Uwe Schindler commented on LUCENE-2791:
---------------------------------------

bq. Sorry, the whole thing was a misunderstanding.

Its ok, we understand. :)

> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Updated: (LUCENE-2791) WindowsDirectory

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Muir updated LUCENE-2791:
--------------------------------

    Attachment: WindowsDirectory_amd64.dll

here's one i compiled for 64-bit. just name it WindowsDirectory.dll and follow the other instructions (obviously, you only use this if you use a 64-bit jvm).

the compilation instructions are in the javadocs for the WindowsDirectory.java by the way, its pretty easy to install mingw or ming64 to do it.


> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Native Policeman (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966486#action_12966486 ] 

Native Policeman commented on LUCENE-2791:
------------------------------------------

There are some violations:

bq. We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

Ahm you dont really use overlapped/async IO, ReadFile in your code only returns when read operation is finished, so not async. Because when you do you have to pass a flag: FILE_FLAG_OVERLAPPED. But I don't think we should really make it async for now, that is too much stuff as Lucene does not really support it at the moment.

- Currently you don't really need the OVERLAPPED structure at all, just pass NULL: [http://msdn.microsoft.com/en-us/library/aa365467(VS.85).aspx], or is there a reason?
- Do we need FILE_SHARE_WRITE?
- You use env->GetStringUTFChars, which is not really a problem as index filenames are always ASCII only. The default methods in the Win32 API use the system default charset (ANSI) for the arguments to OpenFile, which liks to "OpenFileA" in kernel32.dll. I would suggest to use the reaö UTF16 chars (env->GetStringChars) and use OpenFileW method, which takes wchar instead of char. The same applies to Linux Dir, but there I am not sure how encodings are handled.
- Exception handling: in native read, you call throwIOException, when ReadFile() returns false. But 2 lines later you still call the method to copy the local buffer to the java byte array. JNI docs say, that after an Exception is in the thread's local exception, you should not call any JNI methods anymore (expect some allowed ones). Also the copy operation is not really needed.
- As in contract to the native Linux directory we are using our own Handles and not Java's FileDescriptor class to store the handle. If something goes wrong, GC can never clean up the file and it stays open. So you should add a finalizer, Java's internal File methods do that also (RandomAccessFile has a finalizer that calls close on operating system file).

> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966528#action_12966528 ] 

Robert Muir commented on LUCENE-2791:
-------------------------------------

I think, I will also retire as Unicode Policeman, because Unicode is a cruft fore all US americans. Its totally useless!

> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966495#action_12966495 ] 

Robert Muir commented on LUCENE-2791:
-------------------------------------

{quote}
Ahm you dont really use overlapped/async IO, ReadFile in your code only returns when read operation is finished, so not async. Because when you do you have to pass a flag: FILE_FLAG_OVERLAPPED. But I don't think we should really make it async for now, that is too much stuff as Lucene does not really support it at the moment.
{quote}

Yes I do, Overlapped IO is distinct from asynchronous IO! you just use the overlapped structure without OVERLAPPED, as i did here.

bq. Currently you don't really need the OVERLAPPED structure at all, just pass NULL: http://msdn.microsoft.com/en-us/library/aa365467(VS.85).aspx, or is there a reason?

Yes I do use it, do you understand C ?

bq. Do we need FILE_SHARE_WRITE?

Yes

bq. You use env->GetStringUTFChars, which is not really a problem as index filenames are always ASCII only. The default methods in the Win32 API use the system default charset (ANSI) for the arguments to OpenFile, which links to "OpenFileA" in kernel32.dll. I would suggest to use the real UTF16 chars (env->GetStringChars) and use OpenFileW method, which takes wchar instead of char. The same applies to Linux Dir, but there I am not sure how encodings are handled.

We don't open unicode files in lucene, they are all ascii. 

bq. Also the Format functions for GetLastError() results use ANSI per default. We should use the *W method, too, and pass it to java as wchar.

I'm not concerned about this here.

bq. Exception handling: in native read, you call throwIOException, when ReadFile() returns false. But 2 lines later you still call the method to copy the local buffer to the java byte array. JNI docs say, that after an Exception is in the thread's local exception, you should not call any JNI methods anymore (expect some allowed ones). Also the copy operation is not really needed.

I don't think you know what you are talking about!

The code uses 2 techniques, alloc on the stack+SetByteArrayRegion (for small reads), and GetByteArrayElements (for larger reads). The JDK (at least sun, etc) *always* makes a copy for the latter.

So bottom line: I disagree with you on everything you said.


> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Native Policeman (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966505#action_12966505 ] 

Native Policeman commented on LUCENE-2791:
------------------------------------------

bq. But you don't understand how its IO works?

You will laugh, but I did exactly the same thing a few weeks ago :-) Sorry, the whole thing was a misunderstanding.

The other things are simply minor improvements. And Mike also said that we may create files by codecs with names that are non-ascii. And as you are the unicode policeman, I dont unterstand how you can live with default charsets... This is why I suggested to use *W functions in windows and use the UTF16 chars for that reason.

The exception status stuff is also just an optimization in the error case? So why are you aggressive?

> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966487#action_12966487 ] 

Uwe Schindler commented on LUCENE-2791:
---------------------------------------

bq. Does is also allow for deletion of still-open files?

No, of course not. Thats native windos, files that are open cannot be closed. No difference how the file was opened. Also MMapped Files cannot be deleted. Windows does not support the semantics of unix in Win32 API (but it does in POSIX! LOL), so open file: no deleting or changing

> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966512#action_12966512 ] 

Robert Muir commented on LUCENE-2791:
-------------------------------------

bq. Does is also allow for deletion of still-open files?

No, but we could/should investigate trying to improve this, thoguh we might have to implement IndexOutput for it to all work.

bq. You may want to experiment w/ different buffer sizes to see if that impacts performance.

maybe, mainly my point was to avoid the synchronized() here. A user could always tweak the buffer size here?

bq. Also: this allows you to do the equivalent of Linux's O_DIRECT, right? If so... I think we should augment the Dir API so that on opening an input it receives some sort of IOContext or something expressing why we are opening. So for merging we would want "direct iO" (bypass OS buffer cache or NOREUSE fadvise flag, etc.), and larger buffer sizes and perhaps other low level IO settings in the future.

Yes, we can use FILE_FLAG_NO_BUFFERING, though it would be more complicated (things must be sector-aligned). In my opinion this should be a separate IndexInput, and as you stated the Dir API  could inform us. But you are right, the general notion seems to be in all major platforms, and I think we should carefully use it (if it can help!)



> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Updated: (LUCENE-2791) WindowsDirectory

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Muir updated LUCENE-2791:
--------------------------------

    Attachment: LUCENE-2791.patch

i turned mike's benchmark into a unit test, and toned it down a bit.

so i agree with uwe on only one thing, and thats optimizing the case where we throw an exception.

but i disagree on the other things: i think we should just have language neutral error messages and ascii-only files as i presented here... this isn't for general purpose use: its just for lucene. there is no need to bring wchar, etc into this.

so, since all tests pass and this fixes the bug in SimpleFS, i will commit to contrib/misc coon.


> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966508#action_12966508 ] 

Robert Muir commented on LUCENE-2791:
-------------------------------------

bq. And Mike also said that we may create files by codecs with names that are non-ascii. And as you are the unicode policeman, I dont unterstand how you can live with default charsets... 

its my understanding that all indexfiles/codecs are ascii-only.

If this is going to change, then there is a lot of policework to do. Personally i would really prefer if we simply keep codecs and lucene filenames as ascii-only! 

For non-ascii filenames, java.io.File is broken, its equals() is inconsistent with its hashCode(), even on windows, and definitely on things like macos (as i think it still uses unicode normalization to normalize filenames). We should seriously avoid the system-dependent problems that will arise by using non-ascii filenames in these parts of lucene: i don't see this bringing a lot of benefits either.


> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Commented: (LUCENE-2791) WindowsDirectory

Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966489#action_12966489 ] 

Uwe Schindler commented on LUCENE-2791:
---------------------------------------

bq. Also: this allows you to do the equivalent of Linux's O_DIRECT, right? If so... I think we should augment the Dir API so that on opening an input it receives some sort of IOContext or something expressing why we are opening. So for merging we would want "direct iO" (bypass OS buffer cache or NOREUSE fadvise flag, etc.), and larger buffer sizes and perhaps other low level IO settings in the future

Thats different to O_DIRECT. FS caches are still used by windows, but there are also options to disable this: FILE_FLAG_WRITE_THROUGH [http://msdn.microsoft.com/en-us/library/aa363858(v=VS.85).aspx]

> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Updated: (LUCENE-2791) WindowsDirectory

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Muir updated LUCENE-2791:
--------------------------------

    Attachment: LUCENE-2791.patch

attached is an updated patch (fixing some indentation, adding some paranoid NPE checks in the JNI code, various stuff like that)

I also included a crude benchmark made by mikemccand (I think we should commit this to our tests!)

It builds a 100k doc index and searches with 20 threads for 10 seconds:
I used -Dtests.seed=0:0 and -Dtests.codec=Standard, and recorded best out of 3.
I tested win32 -client, and win64 -server, only because these are the oracle defaults.

||Directory||QPS Win32 -client||
|NIOFSDirectory|360|
|SimpleFS|372|
|WindowsDirectory|616|
|RAMDirectory|760|
|MMapDirectory|772|

||Directory||QPS Win64 -server||
|NIOFSDirectory|361|
|SimpleFS|376|
|WindowsDirectory|777|
|RAMDirectory|1105|
|MMapDirectory|1138|

So, I think this gets you past the sync issue and can be a good choice e.g. for win32
But MMap still seems to be the best for this benchmark.



> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch, LUCENE-2791.patch, WindowsDirectory.dll, WindowsDirectory_amd64.dll
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] Updated: (LUCENE-2791) WindowsDirectory

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Muir updated LUCENE-2791:
--------------------------------

    Attachment: LUCENE-2791.patch

here's an initial patch:
All tests pass using this directory with -Dtests.directory, on 32 and 64bit.

I'll attach pre-built DLL files and instructions in case someone else wants to test.


> WindowsDirectory
> ----------------
>
>                 Key: LUCENE-2791
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2791
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Robert Muir
>         Attachments: LUCENE-2791.patch
>
>
> We can use Windows' overlapped IO to do pread() and avoid the performance problems of SimpleFS/NIOFSDir.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org