You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by "Mark Atwell (JIRA)" <ji...@apache.org> on 2005/06/20 17:40:21 UTC
[jira] Created: (DIRMINA-65) Duplicate entries in flush-list, consequent unnecessary system-calls and overzealous wakeups and calls to write(2)
Duplicate entries in flush-list, consequent unnecessary system-calls and overzealous wakeups and calls to write(2)
------------------------------------------------------------------------------------------------------------------
Key: DIRMINA-65
URL: http://issues.apache.org/jira/browse/DIRMINA-65
Project: Directory MINA
Type: Bug
Versions: 0.7.2
Environment: All
Reporter: Mark Atwell
Assigned to: Trustin Lee
The following shows up under heavy load - writing numerous small messages across a slow(ish) network and/or to a slow peer i.e. when a writer starts to see back-log and writes don't necessarily complete.
In SocketIoProcessor:
- scheduleFlush() blindly adds the session to the queue even if it is already present. This means that when the selector thread wakes up it may make numerous calls to (relatively slow) write on the same session.
- flushSession() calls scheduleFlush() and always wakes up the selector thread. This means that flushSessions() and flush() is called for each application write - even when the socket may already be backed up with queued traffic. This is exacerbated by the problem above.
- scheduleFlush() is called from flushSession() which is called from (at least) SocketSessionManagerFilterChain for every application write.
I 'hacked' a test fix which modified scheduleFlush() to only add the session if it wasn't already in the queue and returns a boolean indicating whether the session was added. flushSession() checks this return value and only wakes the selector thread if the session has been added (i.e. it isn't already backed up). In this way writes are not unnecessarily triggered.
This worked though it also highlighted (or brought into greater focus? :o) an apparent timing problem.
java.io.IOException: Invalid argument
at sun.nio.ch.DevPollArrayWrapper.registerMultiple(Native Method)
at sun.nio.ch.DevPollArrayWrapper.updateRegistrations(DevPollArrayWrapper.java:209)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:157)
at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.mina.io.socket.SocketIoProcessor$Worker.run(SocketIoProcessor.java:562)
This isn't fatal though as the retry logic means that the next time 'round the state - presumably of the selection keys - has been brought into alignemnt?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (DIRMINA-65) Duplicate entries in flush-list, consequent unnecessary system-calls and overzealous wakeups and calls to write(2)
Posted by "Trustin Lee (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/DIRMINA-65?page=all ]
Trustin Lee resolved DIRMINA-65:
--------------------------------
Resolution: Fixed
I checked in the fix. I found it is much simpler and efficient to add the checking code to SessionManagerFilterChain implementations.
Could you please try again with the new build and close this issue after testing?
> Duplicate entries in flush-list, consequent unnecessary system-calls and overzealous wakeups and calls to write(2)
> ------------------------------------------------------------------------------------------------------------------
>
> Key: DIRMINA-65
> URL: http://issues.apache.org/jira/browse/DIRMINA-65
> Project: Directory MINA
> Type: Bug
> Versions: 0.7.2, 0.7.1, 0.7
> Environment: All
> Reporter: Mark Atwell
> Assignee: Trustin Lee
> Fix For: 0.7.3
> Attachments: SocketIoProcessor.java
>
> The following shows up under heavy load - writing numerous small messages across a slow(ish) network and/or to a slow peer i.e. when a writer starts to see back-log and writes don't necessarily complete.
> In SocketIoProcessor:
> - scheduleFlush() blindly adds the session to the queue even if it is already present. This means that when the selector thread wakes up it may make numerous calls to (relatively slow) write on the same session.
> - flushSession() calls scheduleFlush() and always wakes up the selector thread. This means that flushSessions() and flush() is called for each application write - even when the socket may already be backed up with queued traffic. This is exacerbated by the problem above.
> - scheduleFlush() is called from flushSession() which is called from (at least) SocketSessionManagerFilterChain for every application write.
> I 'hacked' a test fix which modified scheduleFlush() to only add the session if it wasn't already in the queue and returns a boolean indicating whether the session was added. flushSession() checks this return value and only wakes the selector thread if the session has been added (i.e. it isn't already backed up). In this way writes are not unnecessarily triggered.
> This worked though it also highlighted (or brought into greater focus? :o) an apparent timing problem.
> java.io.IOException: Invalid argument
> at sun.nio.ch.DevPollArrayWrapper.registerMultiple(Native Method)
> at sun.nio.ch.DevPollArrayWrapper.updateRegistrations(DevPollArrayWrapper.java:209)
> at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:157)
> at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.mina.io.socket.SocketIoProcessor$Worker.run(SocketIoProcessor.java:562)
> This isn't fatal though as the retry logic means that the next time 'round the state - presumably of the selection keys - has been brought into alignemnt?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (DIRMINA-65) Duplicate entries in flush-list, consequent unnecessary system-calls and overzealous wakeups and calls to write(2)
Posted by "Trustin Lee (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/DIRMINA-65?page=all ]
Trustin Lee updated DIRMINA-65:
-------------------------------
Fix Version: 0.7.3
Version: 0.7.1
0.7
> Duplicate entries in flush-list, consequent unnecessary system-calls and overzealous wakeups and calls to write(2)
> ------------------------------------------------------------------------------------------------------------------
>
> Key: DIRMINA-65
> URL: http://issues.apache.org/jira/browse/DIRMINA-65
> Project: Directory MINA
> Type: Bug
> Versions: 0.7.2, 0.7.1, 0.7
> Environment: All
> Reporter: Mark Atwell
> Assignee: Trustin Lee
> Fix For: 0.7.3
> Attachments: SocketIoProcessor.java
>
> The following shows up under heavy load - writing numerous small messages across a slow(ish) network and/or to a slow peer i.e. when a writer starts to see back-log and writes don't necessarily complete.
> In SocketIoProcessor:
> - scheduleFlush() blindly adds the session to the queue even if it is already present. This means that when the selector thread wakes up it may make numerous calls to (relatively slow) write on the same session.
> - flushSession() calls scheduleFlush() and always wakes up the selector thread. This means that flushSessions() and flush() is called for each application write - even when the socket may already be backed up with queued traffic. This is exacerbated by the problem above.
> - scheduleFlush() is called from flushSession() which is called from (at least) SocketSessionManagerFilterChain for every application write.
> I 'hacked' a test fix which modified scheduleFlush() to only add the session if it wasn't already in the queue and returns a boolean indicating whether the session was added. flushSession() checks this return value and only wakes the selector thread if the session has been added (i.e. it isn't already backed up). In this way writes are not unnecessarily triggered.
> This worked though it also highlighted (or brought into greater focus? :o) an apparent timing problem.
> java.io.IOException: Invalid argument
> at sun.nio.ch.DevPollArrayWrapper.registerMultiple(Native Method)
> at sun.nio.ch.DevPollArrayWrapper.updateRegistrations(DevPollArrayWrapper.java:209)
> at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:157)
> at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.mina.io.socket.SocketIoProcessor$Worker.run(SocketIoProcessor.java:562)
> This isn't fatal though as the retry logic means that the next time 'round the state - presumably of the selection keys - has been brought into alignemnt?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Closed: (DIRMINA-65) Duplicate entries in flush-list, consequent unnecessary system-calls and overzealous wakeups and calls to write(2)
Posted by "Mark Atwell (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/DIRMINA-65?page=all ]
Mark Atwell closed DIRMINA-65:
------------------------------
> Duplicate entries in flush-list, consequent unnecessary system-calls and overzealous wakeups and calls to write(2)
> ------------------------------------------------------------------------------------------------------------------
>
> Key: DIRMINA-65
> URL: http://issues.apache.org/jira/browse/DIRMINA-65
> Project: Directory MINA
> Type: Bug
> Versions: 0.7.2, 0.7.1, 0.7
> Environment: All
> Reporter: Mark Atwell
> Assignee: Trustin Lee
> Fix For: 0.7.3
> Attachments: SocketIoProcessor.java
>
> The following shows up under heavy load - writing numerous small messages across a slow(ish) network and/or to a slow peer i.e. when a writer starts to see back-log and writes don't necessarily complete.
> In SocketIoProcessor:
> - scheduleFlush() blindly adds the session to the queue even if it is already present. This means that when the selector thread wakes up it may make numerous calls to (relatively slow) write on the same session.
> - flushSession() calls scheduleFlush() and always wakes up the selector thread. This means that flushSessions() and flush() is called for each application write - even when the socket may already be backed up with queued traffic. This is exacerbated by the problem above.
> - scheduleFlush() is called from flushSession() which is called from (at least) SocketSessionManagerFilterChain for every application write.
> I 'hacked' a test fix which modified scheduleFlush() to only add the session if it wasn't already in the queue and returns a boolean indicating whether the session was added. flushSession() checks this return value and only wakes the selector thread if the session has been added (i.e. it isn't already backed up). In this way writes are not unnecessarily triggered.
> This worked though it also highlighted (or brought into greater focus? :o) an apparent timing problem.
> java.io.IOException: Invalid argument
> at sun.nio.ch.DevPollArrayWrapper.registerMultiple(Native Method)
> at sun.nio.ch.DevPollArrayWrapper.updateRegistrations(DevPollArrayWrapper.java:209)
> at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:157)
> at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.mina.io.socket.SocketIoProcessor$Worker.run(SocketIoProcessor.java:562)
> This isn't fatal though as the retry logic means that the next time 'round the state - presumably of the selection keys - has been brought into alignemnt?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DIRMINA-65) Duplicate entries in flush-list, consequent unnecessary system-calls and overzealous wakeups and calls to write(2)
Posted by "Mark Atwell (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/DIRMINA-65?page=comments#action_12314682 ]
Mark Atwell commented on DIRMINA-65:
------------------------------------
No problem, we've all got other things to do! :o)
I haven't investigated the IOException any further. I guess the choice comes down to:
- Whether there is a simple, correct, low-overhead solution - I haven't had time to think about this yet. i.e. what the actual problem is (though I think I'm probably about right), and whether the fix is 'all of the above'.
- The overhead of avoiding an event that, for me at least, so far has been infrequent.
- The risk of not reporting it and accidentally hiding/losing a different more significant problem.
BTW I've logged a bug with Sun about being able to do a ByteBuffer(et al).release().
> Duplicate entries in flush-list, consequent unnecessary system-calls and overzealous wakeups and calls to write(2)
> ------------------------------------------------------------------------------------------------------------------
>
> Key: DIRMINA-65
> URL: http://issues.apache.org/jira/browse/DIRMINA-65
> Project: Directory MINA
> Type: Bug
> Versions: 0.7.2, 0.7.1, 0.7
> Environment: All
> Reporter: Mark Atwell
> Assignee: Trustin Lee
> Fix For: 0.7.3
> Attachments: SocketIoProcessor.java
>
> The following shows up under heavy load - writing numerous small messages across a slow(ish) network and/or to a slow peer i.e. when a writer starts to see back-log and writes don't necessarily complete.
> In SocketIoProcessor:
> - scheduleFlush() blindly adds the session to the queue even if it is already present. This means that when the selector thread wakes up it may make numerous calls to (relatively slow) write on the same session.
> - flushSession() calls scheduleFlush() and always wakes up the selector thread. This means that flushSessions() and flush() is called for each application write - even when the socket may already be backed up with queued traffic. This is exacerbated by the problem above.
> - scheduleFlush() is called from flushSession() which is called from (at least) SocketSessionManagerFilterChain for every application write.
> I 'hacked' a test fix which modified scheduleFlush() to only add the session if it wasn't already in the queue and returns a boolean indicating whether the session was added. flushSession() checks this return value and only wakes the selector thread if the session has been added (i.e. it isn't already backed up). In this way writes are not unnecessarily triggered.
> This worked though it also highlighted (or brought into greater focus? :o) an apparent timing problem.
> java.io.IOException: Invalid argument
> at sun.nio.ch.DevPollArrayWrapper.registerMultiple(Native Method)
> at sun.nio.ch.DevPollArrayWrapper.updateRegistrations(DevPollArrayWrapper.java:209)
> at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:157)
> at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.mina.io.socket.SocketIoProcessor$Worker.run(SocketIoProcessor.java:562)
> This isn't fatal though as the retry logic means that the next time 'round the state - presumably of the selection keys - has been brought into alignemnt?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DIRMINA-65) Duplicate entries in flush-list, consequent unnecessary system-calls and overzealous wakeups and calls to write(2)
Posted by "Trustin Lee (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/DIRMINA-65?page=comments#action_12314625 ]
Trustin Lee commented on DIRMINA-65:
------------------------------------
Thank you for posting a patch for this issue first of all, Mark. I appologize for this late reply. I'm pretty busy these days. :)
Did you get any clue about 'java.io.IOException: Invalid argument' exception message, BTW? If the selector recovers itself automatically in the next loop, it shouldn't be problem. WDYT?
> Duplicate entries in flush-list, consequent unnecessary system-calls and overzealous wakeups and calls to write(2)
> ------------------------------------------------------------------------------------------------------------------
>
> Key: DIRMINA-65
> URL: http://issues.apache.org/jira/browse/DIRMINA-65
> Project: Directory MINA
> Type: Bug
> Versions: 0.7.2, 0.7.1, 0.7
> Environment: All
> Reporter: Mark Atwell
> Assignee: Trustin Lee
> Fix For: 0.7.3
> Attachments: SocketIoProcessor.java
>
> The following shows up under heavy load - writing numerous small messages across a slow(ish) network and/or to a slow peer i.e. when a writer starts to see back-log and writes don't necessarily complete.
> In SocketIoProcessor:
> - scheduleFlush() blindly adds the session to the queue even if it is already present. This means that when the selector thread wakes up it may make numerous calls to (relatively slow) write on the same session.
> - flushSession() calls scheduleFlush() and always wakes up the selector thread. This means that flushSessions() and flush() is called for each application write - even when the socket may already be backed up with queued traffic. This is exacerbated by the problem above.
> - scheduleFlush() is called from flushSession() which is called from (at least) SocketSessionManagerFilterChain for every application write.
> I 'hacked' a test fix which modified scheduleFlush() to only add the session if it wasn't already in the queue and returns a boolean indicating whether the session was added. flushSession() checks this return value and only wakes the selector thread if the session has been added (i.e. it isn't already backed up). In this way writes are not unnecessarily triggered.
> This worked though it also highlighted (or brought into greater focus? :o) an apparent timing problem.
> java.io.IOException: Invalid argument
> at sun.nio.ch.DevPollArrayWrapper.registerMultiple(Native Method)
> at sun.nio.ch.DevPollArrayWrapper.updateRegistrations(DevPollArrayWrapper.java:209)
> at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:157)
> at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.mina.io.socket.SocketIoProcessor$Worker.run(SocketIoProcessor.java:562)
> This isn't fatal though as the retry logic means that the next time 'round the state - presumably of the selection keys - has been brought into alignemnt?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (DIRMINA-65) Duplicate entries in flush-list, consequent unnecessary system-calls and overzealous wakeups and calls to write(2)
Posted by "Mark Atwell (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/DIRMINA-65?page=all ]
Mark Atwell updated DIRMINA-65:
-------------------------------
Attachment: SocketIoProcessor.java
My tentative fix. It has had limited testing. Still has the original SVN ID line in it and some of the whitespace may not be to your taste!
> Duplicate entries in flush-list, consequent unnecessary system-calls and overzealous wakeups and calls to write(2)
> ------------------------------------------------------------------------------------------------------------------
>
> Key: DIRMINA-65
> URL: http://issues.apache.org/jira/browse/DIRMINA-65
> Project: Directory MINA
> Type: Bug
> Versions: 0.7.2
> Environment: All
> Reporter: Mark Atwell
> Assignee: Trustin Lee
> Attachments: SocketIoProcessor.java
>
> The following shows up under heavy load - writing numerous small messages across a slow(ish) network and/or to a slow peer i.e. when a writer starts to see back-log and writes don't necessarily complete.
> In SocketIoProcessor:
> - scheduleFlush() blindly adds the session to the queue even if it is already present. This means that when the selector thread wakes up it may make numerous calls to (relatively slow) write on the same session.
> - flushSession() calls scheduleFlush() and always wakes up the selector thread. This means that flushSessions() and flush() is called for each application write - even when the socket may already be backed up with queued traffic. This is exacerbated by the problem above.
> - scheduleFlush() is called from flushSession() which is called from (at least) SocketSessionManagerFilterChain for every application write.
> I 'hacked' a test fix which modified scheduleFlush() to only add the session if it wasn't already in the queue and returns a boolean indicating whether the session was added. flushSession() checks this return value and only wakes the selector thread if the session has been added (i.e. it isn't already backed up). In this way writes are not unnecessarily triggered.
> This worked though it also highlighted (or brought into greater focus? :o) an apparent timing problem.
> java.io.IOException: Invalid argument
> at sun.nio.ch.DevPollArrayWrapper.registerMultiple(Native Method)
> at sun.nio.ch.DevPollArrayWrapper.updateRegistrations(DevPollArrayWrapper.java:209)
> at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:157)
> at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.mina.io.socket.SocketIoProcessor$Worker.run(SocketIoProcessor.java:562)
> This isn't fatal though as the retry logic means that the next time 'round the state - presumably of the selection keys - has been brought into alignemnt?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira