You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by "Norval Hope (JIRA)" <ji...@apache.org> on 2008/04/14 05:59:04 UTC
[jira] Issue Comment Edited: (DIRSERVER-1161) search results are
not streamed to the client until final done response is queued
[ https://issues.apache.org/jira/browse/DIRSERVER-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588377#action_12588377 ]
nrhope edited comment on DIRSERVER-1161 at 4/13/08 8:58 PM:
-----------------------------------------------------------------
Ok, these attached files represent a patch to ApacheDS 1.5.1 (release tag http://svn.apache.org/repos/asf/directory/apacheds/releases/1.5.1) and latest MINA 1.1 trunk (http://svn.apache.org/repos/asf/mina/branches/1.1 at 647053) which passes the "mvn -Dintegration test" on AD and as well my own test suites working at a higher level).
Note this patch also cover another leak I noticed in SearchHandler when a search returns no results (refer to these lines):
// stop leak of message for each search returning no results
SessionRegistry.getSingleton().removeOutstandingRequest( session, req.getMessageId() );
The changes are discussed in early comments / attachments this issue but are summarized again here:
1. The major problem is that no data is streamed to the JNDI client
(and consequently the queue on the ExecutorFilter.SessionBuffer grows
in an unbounded fashion) until SearchHandler.messageRecieved()
returns. Hence it needs to return as soon as a NamingEnumeration over
the results to be returned is available, meaning a separate thread
needs to be used to drain the SearchResponseIterator asynchronously. Another solution mentioned by Emmanuel would be to force the session to be drained by the client for each result, but I think forcing the server and client to move in such strict lock-step would not be the best approach scalability-wise (and would be harder to implement too).
2. The next problem is the potential for a difference in the client
/ server speed. My patch uses a
LinkedBlockingQueue and this approach worked fine. The problems / notes are:
a. I did my patching against MINA 1.1 trunk (1.1.7-SNAPSHOT) so I
could get Trustin's feedback, but I gather you guys are aiming at
1.1.6 for your upcoming AD 1.5.2 release
b. given Trustin's feedback I'm not sure how "onboard" he is going
to be this the blocking queue approach, but it seems a simple and
clean solution to me
c. configuring the size of the blocking queue (or using a
non-blocking queue if this needs be kept as an
option for other MINA usages) is difficult as SessionBuffer is a static class in
ExecutorFilter and therefore can't be configured via ExecutorFilter's
constructor. I have used a system property as the best solution I can
think of at this stage.
was (Author: nrhope):
Ok, these attached files represent a patch to ApacheDS 1.5.1 (release tag http://svn.apache.org/repos/asf/directory/apacheds/releases/1.5.1) and latest MINA 1.1 trunk (http://svn.apache.org/repos/asf/mina/branches/1.1 at 647053) which passes the "mvn -Dintegration test" on AD and as well my own test suites working at a higher level).
The changes are discussed in early comments / attachments this issue but are summarized again here:
1. The major problem is that no data is streamed to the JNDI client
(and consequently the queue on the ExecutorFilter.SessionBuffer grows
in an unbounded fashion) until SearchHandler.messageRecieved()
returns. Hence it needs to return as soon as a NamingEnumeration over
the results to be returned is available, meaning a separate thread
needs to be used to drain the SearchResponseIterator asynchronously. Another solution mentioned by Emmanuel would be to force the session to be drained by the client for each result, but I think forcing the server and client to move in such strict lock-step would not be the best approach scalability-wise (and would be harder to implement too).
2. The next problem is the potential for a difference in the client
/ server speed. My patch uses a
LinkedBlockingQueue and this approach worked fine. The problems / notes are:
a. I did my patching against MINA 1.1 trunk (1.1.7-SNAPSHOT) so I
could get Trustin's feedback, but I gather you guys are aiming at
1.1.6 for your upcoming AD 1.5.2 release
b. given Trustin's feedback I'm not sure how "onboard" he is going
to be this the blocking queue approach, but it seems a simple and
clean solution to me
c. configuring the size of the blocking queue (or using a
non-blocking queue if this needs be kept as an
option for other MINA usages) is difficult as SessionBuffer is a static class in
ExecutorFilter and therefore can't be configured via ExecutorFilter's
constructor. I have used a system property as the best solution I can
think of at this stage.
> search results are not streamed to the client until final done response is queued
> ---------------------------------------------------------------------------------
>
> Key: DIRSERVER-1161
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1161
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: core
> Affects Versions: 1.0
> Environment: JDK 1.5.0_11
> Reporter: Norval Hope
> Attachments: apacheds_1.5.1_streaming.patch, apacheds_1.5.1_streaming.patch, apacheds_1.5.1_streaming_log_output.txt, installers_1.5.1_streaming.patch, installers_1.5.1_streaming.patch, mina_1.1.2_streaming.patch, mina_1.1_trunk_streaming.patch, pom.xml, streaming_log_output.txt, streaming_logging.patch
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> Search results accumulate in Events on the SessionBuffer.eventQueue within ExecutorFilter.fireEvent() until final done response for the search is written to the session and then all results for the search (possibly millions depending on the search and state of the directory) are written out at once. This is a big problem for scalability and I gather from previous correspondence with Alex that this behaviour is unexpected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.