You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Norval Hope <nr...@gmail.com> on 2008/04/08 15:05:44 UTC
ApacheDS LDAP codec streaming
Hi ApacheDS / MINA-ers,
I am currently doing some load and memory testing against an extension made
to early version of ApacheDS (after it became 1.5 but before a repository
restructure that occurred not long after the rename from 1.0). What I am
finding is that all search result entries are cached in memory due to
ExecutorFilter.fireEvent() queing Events on the session buffer's event queue
that refer to them. Only after the SearchResponseIterator queues the
final SearchResponseDone message and it is encoded and flushed, do any of
the search results actually get flushed out to my JNDI client (JMeter as it
happens). As the searches concerned are attempting to return 100k results
this behaviour leads to a huge heap requirement, and in fact imposes a
seemingly articificial limit on the total number of results which can be
returned by a search (I have gone to great lengths to ensure a "streaming"
approach is used at the level of my custom partitions through the use of
lazy custom NamingEnumerations).
This seems quite bizarre to me as I can see repeated calls to flush encoded
search results but these don't seem to have the effect of streaming search
results incrementally to the client as they become available, which was
my hope / expectation. The streaming behaviour of MINA / ApacheDS was the
subject of a query I posed many months ago and the response from the dev
list at that time was that individual search results would be streamed fine
but that large attributes like JPEGs may pose a problem as they are not
currently streamed. I've spent a full day trying to debug what is going on
between the thread doing the encoding and the thread which recieved the
original search request and then iterated through the search results, but I
still can't see how the SearchResponseDone causes all of the results to
finally be sent when it is eventually written and flushed (and the
synchronize block on the queue is exited) as the same NIO actions of writing
and flushing seem to be applied for all the previous search result messages
too.
Also it seems strange to me that the ExecutorFilter's SessionBuffer
continues to hold references to an Event matching each search result long
after that result was encoded and flushed out. Once the search is 100%
completed and the final search response done message is written, then
the ProcessEventsRunnable.processEvents()
method is called for each of these queued events but actually does nothing
with them.
I was just wanting to check if:
a) this behaviour in the processing of search results has been noticed
before
b) if anyone has any tips on debugging the MINA stack in this particular
case as I've found it extremely hard to get my mind around what is actually
going on re the individual search results which seem to have been flushed
out to NIO but then don't get passed on to the client until the search is
100% finished
as the MINA code concerned is very subtle and I've not tried to groc it
before.
Any help / opinions / pointers gratefully accepted.
Thanks,
Norval
Re: ApacheDS LDAP codec streaming
Posted by Emmanuel Lecharny <el...@gmail.com>.
Hi Norval,
I do have the very same error after having checked out the trunk, and
built the server with a cleared repository... Another bug to fiux ;)
I'll be back to you asap.
Norval Hope wrote:
> As there was no feedback to my initial ping about this streaming
> problem I thought I'd garner more interest by updating a vanilla 1.5
> check-out and see if I can put some debug messages in the right place
> to show the event queue growing by an entry for each search result, if
> indeed the problem is still evident in 1.5. I noticed some directory
> have been restructured when I updated and also found I needed to
> update to at least Maven 2.0.6 so I went for the latest version of
> 2.0.8. <http://2.0.8.>
>
> In the top level "mvn install" using JDK 1.5.0_11 on Windows XP I saw
> a failure with
> ad-1.5-plain\apacheds\protocol-kerberos\src\test\java\org\apache\directory\server\kerberos\protocol\TicketGrantingServiceTest.java
> and moved the test out of the way, however when I re-instated the test
> so I could include the error output it just worked so the failure may
> be intermittent or due to starting with a clean build.
>
> After a bit of poking around the dev list I saw that I now need to run
> ad-1.5-plain\installers\apacheds-noarch\apacheds.bat to get the server
> starting stand-alone, however when I run it I get the following error
> after lots of assembly work has completed:
>
> ==================================================================================================
>
> .....
> [WARNING] Attempting to build MavenProject instance for Artifact
> (org.apache.com <http://org.apache.com>
> mons:commons-io:1.3.2) of type: jar; constructing POM artifact instead.
> [INFO] Expanding: C:\Documents and
> Settings\Norval\.m2\repository\org\apache\com
> mons\commons-io\1.3.2\commons-io-1.3.2.jar into
> C:\DOCUME~1\Norval\LOCALS~1\Temp
> \archived-file-set.107786353.tmp
> [INFO] Expanding: C:\Documents and
> Settings\Norval\.m2\repository\commons-dbcp\c
> ommons-dbcp\1.2.2\commons-dbcp-1.2.2.jar into
> C:\DOCUME~1\Norval\LOCALS~1\Temp\a
> rchived-file-set.2019528138.tmp
> [INFO] Expanding: C:\Documents and
> Settings\Norval\.m2\repository\opensymphony\q
> uartz\1.6.0\quartz-1.6.0.jar into
> C:\DOCUME~1\Norval\LOCALS~1\Temp\archived-file
> -set.777252656.tmp
> [INFO] Building jar:
> C:\src\ad-1.5-plain\installers\apacheds-noarch\target\apach
> eds-noarch-installer2-1.5.2-SNAPSHOT.jar
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] FATAL ERROR
> [INFO]
> ------------------------------------------------------------------------
> [INFO] An invalid artifact was detected.
>
> This artifact might be in your project's POM, or it might have been
> included tra
> nsitively during the resolution process. Here is the information we do
> have for
> this artifact:
>
> o GroupID: org.apache.directory.installers
> o ArtifactID: apacheds-noarch-installer
> o Version: 1.5.2-SNAPSHOT
> o Type: jar
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Trace
> org.apache.maven.artifact.InvalidArtifactRTException: For artifact
> {org.apache.d
> irectory.installers:apacheds-noarch-installer:1.5.2-SNAPSHOT:jar}: An
> attached
> artifact must have a different ID than its corresponding main artifact.
> at
> org.apache.maven.project.artifact.AttachedArtifact.<init>(AttachedArt
> ifact.java:51)
> at
> org.apache.maven.project.DefaultMavenProjectHelper.attachArtifact(Def
> aultMavenProjectHelper.java:53)
> at
> org.apache.maven.plugin.assembly.mojos.AbstractAssemblyMojo.execute(A
> bstractAssemblyMojo.java:295)
> at
> org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPlugi
> nManager.java:447)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(Defa
> ultLifecycleExecutor.java:539)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandalone
> Goal(DefaultLifecycleExecutor.java:493)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(Defau
> ltLifecycleExecutor.java:463)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHan
> dleFailures(DefaultLifecycleExecutor.java:311)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegmen
> ts(DefaultLifecycleExecutor.java:224)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLi
> fecycleExecutor.java:143)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:333)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:126)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:282)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at
> org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
> at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
> at
> org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
>
> at org.codehaus.classworlds.Launcher.main(Launcher.java:375)
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 1 minute 21 seconds
> [INFO] Finished at: Wed Apr 09 12:24:27 EST 2008
> [INFO] Final Memory: 25M/47M
> [INFO]
> ------------------------------------------------------------------------
> Unable to access jarfile
> target/apacheds-noarch-installer-1.5.2-SNAPSHOT-app.jar
>
> ==================================================================================================
>
> Does anyone know what I should do to get around this problem?
>
> Thanks
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: ApacheDS LDAP codec streaming
Posted by Norval Hope <nr...@gmail.com>.
I understand re 1.0. I'm stuck in a backwater on an early 1.5 version due to
a) the AD SVN repository got restructured (seems to happen a lot :-) ) and
I've made quite few customisations that I need to patch across
b) the major part of a) is extensions for loading dynamic schema for
Virtual Directory plug-ins which I have to move over to the new "schema
partition" approach, but in my case I don't want these schemas persisted.
I think it would take me about two weeks to address this task, but as usual
the focus of my managers is on customer facing features rather then trying
to stick to reasonably current versions of the underlying platform. I'm
currently pushing to get time allocated against it for a release around the
end of this year (not so agile :-) ).
If the fact that the JIRA is based on 1.0 is a killer for you guys, then
I'll update tomorrow when you mentioned the trunk would be free of the maven
problem and see if I can apply my minimal patch in the JIRA to the 1.5
structue (plus whichever version of MINA the trunk makes use of these days).
Given my lack of understading of the very complex code, I'm not 100% sure if
the problem is in AD's use of MINA (my presumption) or MINA itself...
Cheers
On Wed, Apr 9, 2008 at 9:53 PM, Emmanuel Lecharny <el...@gmail.com>
wrote:
> Norval Hope wrote:
>
> > Ok, thanks. You will have most probably noticed that I opened
> > DIRSERVER-1161 against 1.0 - the JIRA includes some sample logging output if
> > anyone is interested in pondering it quickly in between beers at some stage
> > :-) Is the fact that it's raised against 1.0 a problem in terms of looking
> > into the issue as you guys have time ? BTW is Trustin at the same conference
> > as you guys ?
> >
> I will download and build 1.0 this afternoon. I ave to warn you that 1.0
> won't be maintained a lot, as we are must more committed in getting a 2.0
> out as soon as possible (lack of time, you know the story ...)
>
>
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>
Re: ApacheDS LDAP codec streaming
Posted by Emmanuel Lecharny <el...@gmail.com>.
Norval Hope wrote:
> Ok, thanks. You will have most probably noticed that I opened
> DIRSERVER-1161 against 1.0 - the JIRA includes some sample logging
> output if anyone is interested in pondering it quickly in between
> beers at some stage :-) Is the fact that it's raised against 1.0 a
> problem in terms of looking into the issue as you guys have time ? BTW
> is Trustin at the same conference as you guys ?
I will download and build 1.0 this afternoon. I ave to warn you that 1.0
won't be maintained a lot, as we are must more committed in getting a
2.0 out as soon as possible (lack of time, you know the story ...)
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: ApacheDS LDAP codec streaming
Posted by Norval Hope <nr...@gmail.com>.
Ok, thanks. You will have most probably noticed that I opened DIRSERVER-1161
against 1.0 - the JIRA includes some sample logging output if anyone is
interested in pondering it quickly in between beers at some stage :-) Is the
fact that it's raised against 1.0 a problem in terms of looking into the
issue as you guys have time ? BTW is Trustin at the same conference as you
guys ?
Cheers
On Wed, Apr 9, 2008 at 8:37 PM, Emmanuel Lecharny <el...@gmail.com>
wrote:
> Norval,
>
> just forget about my previous message, it was done using trunk, which is
> broken atm.
>
> We have fixed the problem a while ago but in a branch, which will become
> the trunk in a few hours, as soon as we have released the 1.5.2 version.
>
> The apacheds.sh script is working in bigbang branch, I just tested it a
> minute ago.
>
>
> Norval Hope wrote:
>
> > As there was no feedback to my initial ping about this streaming problem
> > I thought I'd garner more interest by updating a vanilla 1.5 check-out and
> > see if I can put some debug messages in the right place to show the event
> > queue growing by an entry for each search result, if indeed the problem is
> > still evident in 1.5. I noticed some directory have been restructured when I
> > updated and also found I needed to update to at least Maven 2.0.6 so I went
> > for the latest version of 2.0.8. <http://2.0.8 <http://2.0.0.8/>.>
> >
> > In the top level "mvn install" using JDK 1.5.0_11 on Windows XP I saw a
> > failure with
> > ad-1.5-plain\apacheds\protocol-kerberos\src\test\java\org\apache\directory\server\kerberos\protocol\TicketGrantingServiceTest.java
> > and moved the test out of the way, however when I re-instated the test so I
> > could include the error output it just worked so the failure may be
> > intermittent or due to starting with a clean build.
> >
> > After a bit of poking around the dev list I saw that I now need to run
> > ad-1.5-plain\installers\apacheds-noarch\apacheds.bat to get the server
> > starting stand-alone, however when I run it I get the following error after
> > lots of assembly work has completed:
> >
> >
> > ==================================================================================================
> >
> > .....
> > [WARNING] Attempting to build MavenProject instance for Artifact (
> > org.apache.com <http://org.apache.com>
> >
> > mons:commons-io:1.3.2) of type: jar; constructing POM artifact instead.
> > [INFO] Expanding: C:\Documents and
> > Settings\Norval\.m2\repository\org\apache\com
> > mons\commons-io\1.3.2\commons-io-1.3.2.jar into
> > C:\DOCUME~1\Norval\LOCALS~1\Temp
> > \archived-file-set.107786353.tmp
> > [INFO] Expanding: C:\Documents and
> > Settings\Norval\.m2\repository\commons-dbcp\c
> > ommons-dbcp\1.2.2\commons-dbcp-1.2.2.jar into
> > C:\DOCUME~1\Norval\LOCALS~1\Temp\a
> > rchived-file-set.2019528138.tmp
> > [INFO] Expanding: C:\Documents and
> > Settings\Norval\.m2\repository\opensymphony\q
> > uartz\1.6.0\quartz-1.6.0.jar into
> > C:\DOCUME~1\Norval\LOCALS~1\Temp\archived-file
> > -set.777252656.tmp
> > [INFO] Building jar:
> > C:\src\ad-1.5-plain\installers\apacheds-noarch\target\apach
> > eds-noarch-installer2-1.5.2-SNAPSHOT.jar
> > [INFO]
> > ------------------------------------------------------------------------
> > [ERROR] FATAL ERROR
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] An invalid artifact was detected.
> >
> > This artifact might be in your project's POM, or it might have been
> > included tra
> > nsitively during the resolution process. Here is the information we do
> > have for
> > this artifact:
> >
> > o GroupID: org.apache.directory.installers
> > o ArtifactID: apacheds-noarch-installer
> > o Version: 1.5.2-SNAPSHOT
> > o Type: jar
> >
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] Trace
> > org.apache.maven.artifact.InvalidArtifactRTException: For artifact
> > {org.apache.d
> > irectory.installers:apacheds-noarch-installer:1.5.2-SNAPSHOT:jar}: An
> > attached
> > artifact must have a different ID than its corresponding main artifact.
> > at
> > org.apache.maven.project.artifact.AttachedArtifact.<init>(AttachedArt
> > ifact.java:51)
> > at
> > org.apache.maven.project.DefaultMavenProjectHelper.attachArtifact(Def
> > aultMavenProjectHelper.java:53)
> > at
> > org.apache.maven.plugin.assembly.mojos.AbstractAssemblyMojo.execute(A
> > bstractAssemblyMojo.java:295)
> > at
> > org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPlugi
> > nManager.java:447)
> > at
> > org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(Defa
> > ultLifecycleExecutor.java:539)
> > at
> > org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandalone
> > Goal(DefaultLifecycleExecutor.java:493)
> > at
> > org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(Defau
> > ltLifecycleExecutor.java:463)
> > at
> > org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHan
> > dleFailures(DefaultLifecycleExecutor.java:311)
> > at
> > org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegmen
> > ts(DefaultLifecycleExecutor.java:224)
> > at
> > org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLi
> > fecycleExecutor.java:143)
> > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:333)
> > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:126)
> > at org.apache.maven.cli.MavenCli.main(MavenCli.java:282)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> > java:39)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> > sorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> > org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
> > at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
> > at
> > org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
> >
> > at org.codehaus.classworlds.Launcher.main(Launcher.java:375)
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] Total time: 1 minute 21 seconds
> > [INFO] Finished at: Wed Apr 09 12:24:27 EST 2008
> > [INFO] Final Memory: 25M/47M
> > [INFO]
> > ------------------------------------------------------------------------
> > Unable to access jarfile
> > target/apacheds-noarch-installer-1.5.2-SNAPSHOT-app.jar
> >
> >
> > ==================================================================================================
> >
> > Does anyone know what I should do to get around this problem?
> >
> > Thanks
> >
>
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>
Re: ApacheDS LDAP codec streaming
Posted by Emmanuel Lecharny <el...@gmail.com>.
Norval,
just forget about my previous message, it was done using trunk, which is
broken atm.
We have fixed the problem a while ago but in a branch, which will become
the trunk in a few hours, as soon as we have released the 1.5.2 version.
The apacheds.sh script is working in bigbang branch, I just tested it a
minute ago.
Norval Hope wrote:
> As there was no feedback to my initial ping about this streaming
> problem I thought I'd garner more interest by updating a vanilla 1.5
> check-out and see if I can put some debug messages in the right place
> to show the event queue growing by an entry for each search result, if
> indeed the problem is still evident in 1.5. I noticed some directory
> have been restructured when I updated and also found I needed to
> update to at least Maven 2.0.6 so I went for the latest version of
> 2.0.8. <http://2.0.8.>
>
> In the top level "mvn install" using JDK 1.5.0_11 on Windows XP I saw
> a failure with
> ad-1.5-plain\apacheds\protocol-kerberos\src\test\java\org\apache\directory\server\kerberos\protocol\TicketGrantingServiceTest.java
> and moved the test out of the way, however when I re-instated the test
> so I could include the error output it just worked so the failure may
> be intermittent or due to starting with a clean build.
>
> After a bit of poking around the dev list I saw that I now need to run
> ad-1.5-plain\installers\apacheds-noarch\apacheds.bat to get the server
> starting stand-alone, however when I run it I get the following error
> after lots of assembly work has completed:
>
> ==================================================================================================
>
> .....
> [WARNING] Attempting to build MavenProject instance for Artifact
> (org.apache.com <http://org.apache.com>
> mons:commons-io:1.3.2) of type: jar; constructing POM artifact instead.
> [INFO] Expanding: C:\Documents and
> Settings\Norval\.m2\repository\org\apache\com
> mons\commons-io\1.3.2\commons-io-1.3.2.jar into
> C:\DOCUME~1\Norval\LOCALS~1\Temp
> \archived-file-set.107786353.tmp
> [INFO] Expanding: C:\Documents and
> Settings\Norval\.m2\repository\commons-dbcp\c
> ommons-dbcp\1.2.2\commons-dbcp-1.2.2.jar into
> C:\DOCUME~1\Norval\LOCALS~1\Temp\a
> rchived-file-set.2019528138.tmp
> [INFO] Expanding: C:\Documents and
> Settings\Norval\.m2\repository\opensymphony\q
> uartz\1.6.0\quartz-1.6.0.jar into
> C:\DOCUME~1\Norval\LOCALS~1\Temp\archived-file
> -set.777252656.tmp
> [INFO] Building jar:
> C:\src\ad-1.5-plain\installers\apacheds-noarch\target\apach
> eds-noarch-installer2-1.5.2-SNAPSHOT.jar
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] FATAL ERROR
> [INFO]
> ------------------------------------------------------------------------
> [INFO] An invalid artifact was detected.
>
> This artifact might be in your project's POM, or it might have been
> included tra
> nsitively during the resolution process. Here is the information we do
> have for
> this artifact:
>
> o GroupID: org.apache.directory.installers
> o ArtifactID: apacheds-noarch-installer
> o Version: 1.5.2-SNAPSHOT
> o Type: jar
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Trace
> org.apache.maven.artifact.InvalidArtifactRTException: For artifact
> {org.apache.d
> irectory.installers:apacheds-noarch-installer:1.5.2-SNAPSHOT:jar}: An
> attached
> artifact must have a different ID than its corresponding main artifact.
> at
> org.apache.maven.project.artifact.AttachedArtifact.<init>(AttachedArt
> ifact.java:51)
> at
> org.apache.maven.project.DefaultMavenProjectHelper.attachArtifact(Def
> aultMavenProjectHelper.java:53)
> at
> org.apache.maven.plugin.assembly.mojos.AbstractAssemblyMojo.execute(A
> bstractAssemblyMojo.java:295)
> at
> org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPlugi
> nManager.java:447)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(Defa
> ultLifecycleExecutor.java:539)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandalone
> Goal(DefaultLifecycleExecutor.java:493)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(Defau
> ltLifecycleExecutor.java:463)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHan
> dleFailures(DefaultLifecycleExecutor.java:311)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegmen
> ts(DefaultLifecycleExecutor.java:224)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLi
> fecycleExecutor.java:143)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:333)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:126)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:282)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at
> org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
> at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
> at
> org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
>
> at org.codehaus.classworlds.Launcher.main(Launcher.java:375)
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 1 minute 21 seconds
> [INFO] Finished at: Wed Apr 09 12:24:27 EST 2008
> [INFO] Final Memory: 25M/47M
> [INFO]
> ------------------------------------------------------------------------
> Unable to access jarfile
> target/apacheds-noarch-installer-1.5.2-SNAPSHOT-app.jar
>
> ==================================================================================================
>
> Does anyone know what I should do to get around this problem?
>
> Thanks
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: ApacheDS LDAP codec streaming
Posted by Norval Hope <nr...@gmail.com>.
Thanks for the response - enjoy your presentations - and even more so your
beer!
Rather the persist with the maven problem observed with 1.5 I'm now trying
out against
http://svn.apache.org/repos/asf/directory/apacheds/branches/1.0-with-dependenciesand
I'm adding some extra log messages in SearchHandler and MINA's
ExecutorFilter for which I'll provide a patch. I'm currently doing runs with
the extra logging to ensure they make the problem clear and will then apply
the patch to the vanilla 1.0 and check that the problem is still observable
on it too. If so then I'll open a JIRA and include the extra debugging patch
plus a bunch of observations of stuff I've noticed in the code that looks a
bit weird, but that may be a result of me not fully understanding how all
the MINA stuff hangs together in AD.
Also I noticed a memory leak in SearchHandler when a search returns no
results which I rectified using the "nhope" commented line below:
if ( list.hasMore() )
{
Iterator it = new SearchResponseIterator( req, ctx, list,
controls.getSearchScope(), session );
while ( it.hasNext() )
{
final Object next = it.next();
if ( IS_DEBUG )
{
log.debug( "*** Search: WRITING to session: " +
session.getRemoteAddress() + ": object=" + next);
}
session.write( next );
}
return;
}
else
{
list.close();
req.getResultResponse().getLdapResult().setResultCode(
ResultCodeEnum.SUCCESS );
Iterator it = Collections.singleton( req.getResultResponse()
).iterator();
while ( it.hasNext() )
{
session.write( it.next() );
}
// nhope: seem to leak requests in registry for empty
searches otherwise
SessionRegistry.getSingleton().removeOutstandingRequest(
session, req.getMessageId() );
return;
}
Cheers,
Norval
Re: ApacheDS LDAP codec streaming
Posted by Emmanuel Lecharny <el...@gmail.com>.
Norval Hope wrote:
> As there was no feedback to my initial ping about this streaming
> problem I thought I'd garner more interest
Hi Norval,
sorry for the delay, we were all moving to Apache conference in
Masterdam, so we didn't had time to check all our emails...
> by updating a vanilla 1.5 check-out and see if I can put some debug
> messages in the right place to show the event queue growing by an
> entry for each search result, if indeed the problem is still evident
> in 1.5. I noticed some directory have been restructured when I updated
> and also found I needed to update to at least Maven 2.0.6 so I went
> for the latest version of 2.0.8. <http://2.0.8.>
Thanks for having spent time to provide more information !
>
> In the top level "mvn install" using JDK 1.5.0_11 on Windows XP I saw
> a failure with
> ad-1.5-plain\apacheds\protocol-kerberos\src\test\java\org\apache\directory\server\kerberos\protocol\TicketGrantingServiceTest.java
> and moved the test out of the way, however when I re-instated the test
> so I could include the error output it just worked so the failure may
> be intermittent or due to starting with a clean build.
>
> After a bit of poking around the dev list I saw that I now need to run
> ad-1.5-plain\installers\apacheds-noarch\apacheds.bat to get the server
> starting stand-alone, however when I run it I get the following error
> after lots of assembly work has completed:
>
> ==================================================================================================
>
> .....
> [WARNING] Attempting to build MavenProject instance for Artifact
> (org.apache.com <http://org.apache.com>
> mons:commons-io:1.3.2) of type: jar; constructing POM artifact instead.
> [INFO] Expanding: C:\Documents and
> Settings\Norval\.m2\repository\org\apache\com
> mons\commons-io\1.3.2\commons-io-1.3.2.jar into
> C:\DOCUME~1\Norval\LOCALS~1\Temp
> \archived-file-set.107786353.tmp
> [INFO] Expanding: C:\Documents and
> Settings\Norval\.m2\repository\commons-dbcp\c
> ommons-dbcp\1.2.2\commons-dbcp-1.2.2.jar into
> C:\DOCUME~1\Norval\LOCALS~1\Temp\a
> rchived-file-set.2019528138.tmp
> [INFO] Expanding: C:\Documents and
> Settings\Norval\.m2\repository\opensymphony\q
> uartz\1.6.0\quartz-1.6.0.jar into
> C:\DOCUME~1\Norval\LOCALS~1\Temp\archived-file
> -set.777252656.tmp
> [INFO] Building jar:
> C:\src\ad-1.5-plain\installers\apacheds-noarch\target\apach
> eds-noarch-installer2-1.5.2-SNAPSHOT.jar
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] FATAL ERROR
> [INFO]
> ------------------------------------------------------------------------
> [INFO] An invalid artifact was detected.
>
> This artifact might be in your project's POM, or it might have been
> included tra
> nsitively during the resolution process. Here is the information we do
> have for
> this artifact:
>
> o GroupID: org.apache.directory.installers
> o ArtifactID: apacheds-noarch-installer
> o Version: 1.5.2-SNAPSHOT
> o Type: jar
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Trace
> org.apache.maven.artifact.InvalidArtifactRTException: For artifact
> {org.apache.d
> irectory.installers:apacheds-noarch-installer:1.5.2-SNAPSHOT:jar}: An
> attached
> artifact must have a different ID than its corresponding main artifact.
> at
> org.apache.maven.project.artifact.AttachedArtifact.<init>(AttachedArt
> ifact.java:51)
> at
> org.apache.maven.project.DefaultMavenProjectHelper.attachArtifact(Def
> aultMavenProjectHelper.java:53)
> at
> org.apache.maven.plugin.assembly.mojos.AbstractAssemblyMojo.execute(A
> bstractAssemblyMojo.java:295)
> at
> org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPlugi
> nManager.java:447)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(Defa
> ultLifecycleExecutor.java:539)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandalone
> Goal(DefaultLifecycleExecutor.java:493)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(Defau
> ltLifecycleExecutor.java:463)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHan
> dleFailures(DefaultLifecycleExecutor.java:311)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegmen
> ts(DefaultLifecycleExecutor.java:224)
> at
> org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLi
> fecycleExecutor.java:143)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:333)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:126)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:282)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at
> org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
> at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
> at
> org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
>
> at org.codehaus.classworlds.Launcher.main(Launcher.java:375)
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 1 minute 21 seconds
> [INFO] Finished at: Wed Apr 09 12:24:27 EST 2008
> [INFO] Final Memory: 25M/47M
> [INFO]
> ------------------------------------------------------------------------
> Unable to access jarfile
> target/apacheds-noarch-installer-1.5.2-SNAPSHOT-app.jar
>
> ==================================================================================================
>
> Does anyone know what I should do to get around this problem?
Gimme a few time so that I can check on my computer if I get the same
problem. May I suggest you try to run the
/project/resources/superclean.sh script, rebuild the full project with a
mvn clean install and try to run the apacheds.sh script ? You may are
using an old version of some jars, and the superclean.sh will remove all
the apacheds jars from your local maven repository.
In any case, we are just releasing ADS 1.5.2 today.
Be ready to get some latency from us as we are all pretty busy with
presentations, talks - and beers :) - but we will do our best to help
you withn the problem, considereing that we if the server has the
problem you mentionned, then it's a *big* bug.
Thanks Norval !
>
> Thanks
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: ApacheDS LDAP codec streaming
Posted by Norval Hope <nr...@gmail.com>.
As there was no feedback to my initial ping about this streaming problem I
thought I'd garner more interest by updating a vanilla 1.5 check-out and see
if I can put some debug messages in the right place to show the event queue
growing by an entry for each search result, if indeed the problem is still
evident in 1.5. I noticed some directory have been restructured when I
updated and also found I needed to update to at least Maven 2.0.6 so I went
for the latest version of 2.0.8.
In the top level "mvn install" using JDK 1.5.0_11 on Windows XP I saw a
failure with
ad-1.5-plain\apacheds\protocol-kerberos\src\test\java\org\apache\directory\server\kerberos\protocol\TicketGrantingServiceTest.java
and moved the test out of the way, however when I re-instated the test so I
could include the error output it just worked so the failure may be
intermittent or due to starting with a clean build.
After a bit of poking around the dev list I saw that I now need to run
ad-1.5-plain\installers\apacheds-noarch\apacheds.bat to get the server
starting stand-alone, however when I run it I get the following error after
lots of assembly work has completed:
==================================================================================================
.....
[WARNING] Attempting to build MavenProject instance for Artifact (
org.apache.com
mons:commons-io:1.3.2) of type: jar; constructing POM artifact instead.
[INFO] Expanding: C:\Documents and
Settings\Norval\.m2\repository\org\apache\com
mons\commons-io\1.3.2\commons-io-1.3.2.jar into
C:\DOCUME~1\Norval\LOCALS~1\Temp
\archived-file-set.107786353.tmp
[INFO] Expanding: C:\Documents and
Settings\Norval\.m2\repository\commons-dbcp\c
ommons-dbcp\1.2.2\commons-dbcp-1.2.2.jar into
C:\DOCUME~1\Norval\LOCALS~1\Temp\a
rchived-file-set.2019528138.tmp
[INFO] Expanding: C:\Documents and
Settings\Norval\.m2\repository\opensymphony\q
uartz\1.6.0\quartz-1.6.0.jar into
C:\DOCUME~1\Norval\LOCALS~1\Temp\archived-file
-set.777252656.tmp
[INFO] Building jar:
C:\src\ad-1.5-plain\installers\apacheds-noarch\target\apach
eds-noarch-installer2-1.5.2-SNAPSHOT.jar
[INFO]
------------------------------------------------------------------------
[ERROR] FATAL ERROR
[INFO]
------------------------------------------------------------------------
[INFO] An invalid artifact was detected.
This artifact might be in your project's POM, or it might have been included
tra
nsitively during the resolution process. Here is the information we do have
for
this artifact:
o GroupID: org.apache.directory.installers
o ArtifactID: apacheds-noarch-installer
o Version: 1.5.2-SNAPSHOT
o Type: jar
[INFO]
------------------------------------------------------------------------
[INFO] Trace
org.apache.maven.artifact.InvalidArtifactRTException: For artifact
{org.apache.d
irectory.installers:apacheds-noarch-installer:1.5.2-SNAPSHOT:jar}: An
attached
artifact must have a different ID than its corresponding main artifact.
at
org.apache.maven.project.artifact.AttachedArtifact.<init>(AttachedArt
ifact.java:51)
at
org.apache.maven.project.DefaultMavenProjectHelper.attachArtifact(Def
aultMavenProjectHelper.java:53)
at
org.apache.maven.plugin.assembly.mojos.AbstractAssemblyMojo.execute(A
bstractAssemblyMojo.java:295)
at
org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPlugi
nManager.java:447)
at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(Defa
ultLifecycleExecutor.java:539)
at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandalone
Goal(DefaultLifecycleExecutor.java:493)
at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(Defau
ltLifecycleExecutor.java:463)
at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHan
dleFailures(DefaultLifecycleExecutor.java:311)
at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegmen
ts(DefaultLifecycleExecutor.java:224)
at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLi
fecycleExecutor.java:143)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:333)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:126)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:282)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
at
org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
at org.codehaus.classworlds.Launcher.main(Launcher.java:375)
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 1 minute 21 seconds
[INFO] Finished at: Wed Apr 09 12:24:27 EST 2008
[INFO] Final Memory: 25M/47M
[INFO]
------------------------------------------------------------------------
Unable to access jarfile
target/apacheds-noarch-installer-1.5.2-SNAPSHOT-app.jar
==================================================================================================
Does anyone know what I should do to get around this problem?
Thanks
Re: ApacheDS LDAP codec streaming
Posted by Norval Hope <nr...@gmail.com>.
I should also mention that I'm using JDK 1.5.0_11.
On Tue, Apr 8, 2008 at 11:09 PM, Norval Hope <nr...@gmail.com> wrote:
> As a side note here is an excerpt of the old thread which I mentioned with
> a question from me and an answer from Alex:
>
> NH> I was just wanting to ask one of the ADS gurus what happens exactly at
> NH> the LDAP protocol level when an extremely large result set is returned
> NH> by ADS for an LDAP search(). Is a client able to start retreiving
> NH> results via their NamingEnumeration before the server has completed
> NH> its search? I'm hoping there is some sort of "chunking" at the
> NH> protocol level so that batches of results are streamed to the client
> NH> as they become available, rather then the waiting for the entire
> NH> result set to be available first. Is this the case?
>
> Yes and no :). There is no chunking of an individual PDU sent back.
> Meaning each SearchEntryResponse prepared then sent out the door. If an
> entry you're returning has a 1 mb jpeg in it then this will be used to
> prepare a response and return it rather than streaming out the jpeg. Old
> incarnations of MINA used to allow us to do this but it was complicated.
> For the search operation, we do not collect all entries before sending a
> response. That would kill the server. ApacheDS will pull out and send a
> result entry to you one at a time. MINA may lag a little bit and some
> results may actually sit a while before going out the door. But for all
> practical purposes we pull from the db and pump out results one by one.
> The way it works is a bit complicated and will take some time to discuss.
> I can go into it if enough people are interested and willing to document it
> :) after I describe how she does this.
> Alex
>
> On Tue, Apr 8, 2008 at 11:05 PM, Norval Hope <nr...@gmail.com> wrote:
>
> > Hi ApacheDS / MINA-ers,
> >
> > I am currently doing some load and memory testing against an extension
> > made to early version of ApacheDS (after it became 1.5 but before a
> > repository restructure that occurred not long after the rename from 1.0).
> > What I am finding is that all search result entries are cached in memory due
> > to ExecutorFilter.fireEvent() queing Events on the session buffer's event
> > queue that refer to them. Only after the SearchResponseIterator queues the
> > final SearchResponseDone message and it is encoded and flushed, do any of
> > the search results actually get flushed out to my JNDI client (JMeter as it
> > happens). As the searches concerned are attempting to return 100k results
> > this behaviour leads to a huge heap requirement, and in fact imposes a
> > seemingly articificial limit on the total number of results which can be
> > returned by a search (I have gone to great lengths to ensure a "streaming"
> > approach is used at the level of my custom partitions through the use of
> > lazy custom NamingEnumerations).
> >
> > This seems quite bizarre to me as I can see repeated calls to flush
> > encoded search results but these don't seem to have the effect of streaming
> > search results incrementally to the client as they become available, which
> > was my hope / expectation. The streaming behaviour of MINA / ApacheDS was
> > the subject of a query I posed many months ago and the response from the dev
> > list at that time was that individual search results would be streamed fine
> > but that large attributes like JPEGs may pose a problem as they are not
> > currently streamed. I've spent a full day trying to debug what is going on
> > between the thread doing the encoding and the thread which recieved the
> > original search request and then iterated through the search results, but I
> > still can't see how the SearchResponseDone causes all of the results to
> > finally be sent when it is eventually written and flushed (and the
> > synchronize block on the queue is exited) as the same NIO actions of writing
> > and flushing seem to be applied for all the previous search result messages
> > too.
> >
> > Also it seems strange to me that the ExecutorFilter's SessionBuffer
> > continues to hold references to an Event matching each search result long
> > after that result was encoded and flushed out. Once the search is 100%
> > completed and the final search response done message is written, then the ProcessEventsRunnable.processEvents()
> > method is called for each of these queued events but actually does nothing
> > with them.
> >
> > I was just wanting to check if:
> >
> > a) this behaviour in the processing of search results has been noticed
> > before
> > b) if anyone has any tips on debugging the MINA stack in this
> > particular case as I've found it extremely hard to get my mind around what
> > is actually going on re the individual search results which seem to have
> > been flushed out to NIO but then don't get passed on to the client until the
> > search is 100% finished
> >
> > as the MINA code concerned is very subtle and I've not tried to groc it
> > before.
> >
> > Any help / opinions / pointers gratefully accepted.
> >
> > Thanks,
> > Norval
> >
>
>
Re: ApacheDS LDAP codec streaming
Posted by Norval Hope <nr...@gmail.com>.
As a side note here is an excerpt of the old thread which I mentioned with a
question from me and an answer from Alex:
NH> I was just wanting to ask one of the ADS gurus what happens exactly at
NH> the LDAP protocol level when an extremely large result set is returned
NH> by ADS for an LDAP search(). Is a client able to start retreiving
NH> results via their NamingEnumeration before the server has completed
NH> its search? I'm hoping there is some sort of "chunking" at the
NH> protocol level so that batches of results are streamed to the client
NH> as they become available, rather then the waiting for the entire
NH> result set to be available first. Is this the case?
Yes and no :). There is no chunking of an individual PDU sent back. Meaning
each SearchEntryResponse prepared then sent out the door. If an entry
you're returning has a 1 mb jpeg in it then this will be used to prepare a
response and return it rather than streaming out the jpeg. Old incarnations
of MINA used to allow us to do this but it was complicated.
For the search operation, we do not collect all entries before sending a
response. That would kill the server. ApacheDS will pull out and send a
result entry to you one at a time. MINA may lag a little bit and some
results may actually sit a while before going out the door. But for all
practical purposes we pull from the db and pump out results one by one.
The way it works is a bit complicated and will take some time to discuss. I
can go into it if enough people are interested and willing to document it :)
after I describe how she does this.
Alex
On Tue, Apr 8, 2008 at 11:05 PM, Norval Hope <nr...@gmail.com> wrote:
> Hi ApacheDS / MINA-ers,
>
> I am currently doing some load and memory testing against an extension
> made to early version of ApacheDS (after it became 1.5 but before a
> repository restructure that occurred not long after the rename from 1.0).
> What I am finding is that all search result entries are cached in memory due
> to ExecutorFilter.fireEvent() queing Events on the session buffer's event
> queue that refer to them. Only after the SearchResponseIterator queues the
> final SearchResponseDone message and it is encoded and flushed, do any of
> the search results actually get flushed out to my JNDI client (JMeter as it
> happens). As the searches concerned are attempting to return 100k results
> this behaviour leads to a huge heap requirement, and in fact imposes a
> seemingly articificial limit on the total number of results which can be
> returned by a search (I have gone to great lengths to ensure a "streaming"
> approach is used at the level of my custom partitions through the use of
> lazy custom NamingEnumerations).
>
> This seems quite bizarre to me as I can see repeated calls to flush
> encoded search results but these don't seem to have the effect of streaming
> search results incrementally to the client as they become available, which
> was my hope / expectation. The streaming behaviour of MINA / ApacheDS was
> the subject of a query I posed many months ago and the response from the dev
> list at that time was that individual search results would be streamed fine
> but that large attributes like JPEGs may pose a problem as they are not
> currently streamed. I've spent a full day trying to debug what is going on
> between the thread doing the encoding and the thread which recieved the
> original search request and then iterated through the search results, but I
> still can't see how the SearchResponseDone causes all of the results to
> finally be sent when it is eventually written and flushed (and the
> synchronize block on the queue is exited) as the same NIO actions of writing
> and flushing seem to be applied for all the previous search result messages
> too.
>
> Also it seems strange to me that the ExecutorFilter's SessionBuffer
> continues to hold references to an Event matching each search result long
> after that result was encoded and flushed out. Once the search is 100%
> completed and the final search response done message is written, then the ProcessEventsRunnable.processEvents()
> method is called for each of these queued events but actually does nothing
> with them.
>
> I was just wanting to check if:
>
> a) this behaviour in the processing of search results has been noticed
> before
> b) if anyone has any tips on debugging the MINA stack in this particular
> case as I've found it extremely hard to get my mind around what is actually
> going on re the individual search results which seem to have been flushed
> out to NIO but then don't get passed on to the client until the search is
> 100% finished
>
> as the MINA code concerned is very subtle and I've not tried to groc it
> before.
>
> Any help / opinions / pointers gratefully accepted.
>
> Thanks,
> Norval
>