[jira] Commented: (DIRSERVER-1161) search results are not streamed to the client until final done response is queued

["Norval Hope (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DIRSERVER-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587092#action_12587092 ] 

Norval Hope commented on DIRSERVER-1161:
----------------------------------------

Other things I've noticed in analysing this issue (these may or may not be knock-on problems which tie back to the lack of streaming):
  1. MINA's ProtocolCodecFilter has the following inner class where there are no accessors for "message" which leaves me thinking this object is for debugging only in which case it maybe needs to be made conditional

   private static class MessageByteBuffer extends ByteBufferProxy
    {
        private final Object message;
  2. In general when I look at memory usage I see a ton of strings which are RDN and LdapDN instances that it seems to me could be better handled better using String.intern()
  3. In SearchResponseIterator I see this code and wonder why the LdapDN needs to be created at @@@@@ for each search result (if this is just testing for validity does it make sense to make this behaviour configurable via server.xml):

if ( !isReferral || req.getControls().containsKey( ManageDsaITControl.CONTROL_OID ) )
        {
            SearchResponseEntry respEntry = new SearchResponseEntryImpl( req.getMessageId() );
            respEntry.setAttributes( result.getAttributes() );
            
            try
            {
                respEntry.setObjectName( new LdapDN( result.getName() ) ); // @@@@@@
            }
            catch ( InvalidNameException ine )
            {
                log.error( "Invalid object name : " + result.getName(), ine);
                throw new RuntimeException( ine );
            }
            
            prefetched = respEntry;
        }

   4. Even if incremental streaming is achieved I'm worried that the architecture doesn't seem to cope with the case where SearchHandler generates results quicker then the JNDI client consumes them, so that an unbounded amount of encoded results can build up on the server waiting to be transmitted, or does NIO take care of this (assuming the current streaming bug is fixed) by blocking the server in some way?

> search results are not streamed to the client until final done response is queued
> ---------------------------------------------------------------------------------
>
>                 Key: DIRSERVER-1161
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-1161
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 1.0
>         Environment: JDK 1.5.0_11 
>            Reporter: Norval Hope
>         Attachments: streaming_log_output.txt, streaming_logging.patch
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> Search results accumulate in Events on the SessionBuffer.eventQueue within ExecutorFilter.fireEvent() until final done response for the search is written to the session and then all results for the search (possibly millions depending on the search and state of the directory) are written out at once. This is a big problem for scalability and I gather from previous correspondence with Alex that this behaviour is unexpected.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.