You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Kevin Smith <ks...@redhat.com> on 2007/05/16 14:48:48 UTC
C++ broker issues?
I've heard that the C++ broker has concurrency issues -- can anyone confirm
this? I'm running into a situation where more than two concurrent clients cause
the broker to wedge itself at 100% CPU and never return.
--Kevin
Re: M2 ?
Posted by Rafael Schloming <ra...@redhat.com>.
Marnie McCormack wrote:
> Hi Rafi,
>
> Do the python and ruby tests include the kind of simple interop stuff that
> Rupert's been working on for the other implementations ?
Are you referring to the kinds of tests described here?
http://cwiki.apache.org/confluence/display/qpid/Interop+Testing+Specification
I'm afraid I haven't been following that discussion closely. Based on my
quick read it looks like they do test the kinds of things outlined on
that page, e.g. basic publish and consume. Most of the tests, however,
are focused on detailed broker behavior, not interop between different
kinds of clients.
Alan or Gordon may be able to answer your question better.
--Rafael
>
> Thanks & Regards,
> Marnie
>
>
> On 5/25/07, Rafael Schloming <ra...@redhat.com> wrote:
>>
>> Kevin Smith wrote:
>> > Robert Godfrey wrote:
>> >> Anyone want to volunteer to take on the Python or Ruby?
>> >>
>> >> -- Rob
>> >
>> > I might, depending on what's involved. Could someone spell out what
>> > needs to be done here?
>>
>> Code-wise there isn't really anything required for M2 on python or ruby.
>> When the question arose back when on whether or not to include ruby I
>> added some basic test coverage for ruby and fixed a number of issues
>> that came up.
>>
>> Integration-wise we of course need to make sure the ruby and python
>> tests pass against the java and cpp brokers. I just checked the python
>> tests against the java broker and they all pass. There is one failure in
>> the ruby suite. I haven't looked at it in detail. Both test suites
>> should of course be run against the cpp broker as well.
>>
>> --Rafael
>>
>
Re: M2 ?
Posted by Marnie McCormack <ma...@googlemail.com>.
Hi Rafi,
Do the python and ruby tests include the kind of simple interop stuff that
Rupert's been working on for the other implementations ?
Thanks & Regards,
Marnie
On 5/25/07, Rafael Schloming <ra...@redhat.com> wrote:
>
> Kevin Smith wrote:
> > Robert Godfrey wrote:
> >> Anyone want to volunteer to take on the Python or Ruby?
> >>
> >> -- Rob
> >
> > I might, depending on what's involved. Could someone spell out what
> > needs to be done here?
>
> Code-wise there isn't really anything required for M2 on python or ruby.
> When the question arose back when on whether or not to include ruby I
> added some basic test coverage for ruby and fixed a number of issues
> that came up.
>
> Integration-wise we of course need to make sure the ruby and python
> tests pass against the java and cpp brokers. I just checked the python
> tests against the java broker and they all pass. There is one failure in
> the ruby suite. I haven't looked at it in detail. Both test suites
> should of course be run against the cpp broker as well.
>
> --Rafael
>
Re: M2 ?
Posted by Rafael Schloming <ra...@redhat.com>.
Kevin Smith wrote:
> Robert Godfrey wrote:
>> Anyone want to volunteer to take on the Python or Ruby?
>>
>> -- Rob
>
> I might, depending on what's involved. Could someone spell out what
> needs to be done here?
Code-wise there isn't really anything required for M2 on python or ruby.
When the question arose back when on whether or not to include ruby I
added some basic test coverage for ruby and fixed a number of issues
that came up.
Integration-wise we of course need to make sure the ruby and python
tests pass against the java and cpp brokers. I just checked the python
tests against the java broker and they all pass. There is one failure in
the ruby suite. I haven't looked at it in detail. Both test suites
should of course be run against the cpp broker as well.
--Rafael
Re: M2 ?
Posted by Kevin Smith <ks...@redhat.com>.
Robert Godfrey wrote:
> Anyone want to volunteer to take on the Python or Ruby?
>
> -- Rob
I might, depending on what's involved. Could someone spell out what needs to be
done here?
--Kevin
Re: M2 ?
Posted by Robert Godfrey <ro...@gmail.com>.
Anyone want to volunteer to take on the Python or Ruby?
-- Rob
On 24/05/07, Rupert Smith <ru...@googlemail.com> wrote:
> Interop tests available on Java and C++, a .Net port is in the pipeline but
> not started yet.
>
> On 24/05/07, Marnie McCormack <ma...@googlemail.com> wrote:
> >
> > I guess the more 'how long is a piece of string' like question is around
> > interop ? (Sorry) I'm a bit detached from progress that Rupert's been
> > making
> > on that front ?
> >
> > Also, there's the python and ruby clients. On account of the votes (and my
> > ignorance of the binding process with components in votes for releases) we
> > are currently compelled to include them. I guess that means they need to
> > work well with both brokers, or we vote to reduce the scope of the
> > release,
> > as well as be doc'd enough for users ?
> >
> > Bfn,
> > Regards,
> > Marnie
> >
> >
> > On 5/24/07, Martin Ritchie <ri...@apache.org> wrote:
> > >
> > > I've been trying to tidy up the Java side also. With some recent
> > > functional additions but mainly it has been fixing the odd bug. There
> > > are a couple of additions that should be JIRAed that would be good to
> > > go in with M2 such as the Mina upgrade(QPID-92) along with limiting
> > > the size of our send/receive buffers so we reduce the OOM issues with
> > > mina, and QPID-465 for JMS compliance.
> > >
> > > We also have a bug in the client where FailoverExceptions can 'escape'
> > > as not all client api calls correctly handle failover,QPID-402.
> > >
> > > The other serious issue is an un-JIRAed item about our implementation
> > > of AMQMessage on the server side. When used in Pub Sub or delivered to
> > > multiple queues then looks like we have a problem.
> > >
> > > Fixing those along with my current work load will probably take us to
> > mid
> > > June.
> > >
> > > On 23/05/07, Gordon Sim <gs...@redhat.com> wrote:
> > > > Rajith Attapattu wrote:
> > > > > Alan can u comment on the c++ side. If I am not mistaken Alan havwe
> > > > > already
> > > > > identified JIRA's for M2 and is in the process of fixing them.
> > > >
> > > > Alan is on holiday this week. However we have a fix to the apr_pool
> > > > related concurrency issue, channel.flow is implemented and the interop
> > > > tests are written. That more or less covers the open JIRAs I think...
> > a
> > > > couple of them are still marked open as I haven't merged back to trunk
> > > > yet (due to other stuff I've got on the go on trunk!).
> > > >
> > >
> > >
> > > --
> > > Martin Ritchie
> > >
> >
>
Re: M2 ?
Posted by Rupert Smith <ru...@googlemail.com>.
Interop tests available on Java and C++, a .Net port is in the pipeline but
not started yet.
On 24/05/07, Marnie McCormack <ma...@googlemail.com> wrote:
>
> I guess the more 'how long is a piece of string' like question is around
> interop ? (Sorry) I'm a bit detached from progress that Rupert's been
> making
> on that front ?
>
> Also, there's the python and ruby clients. On account of the votes (and my
> ignorance of the binding process with components in votes for releases) we
> are currently compelled to include them. I guess that means they need to
> work well with both brokers, or we vote to reduce the scope of the
> release,
> as well as be doc'd enough for users ?
>
> Bfn,
> Regards,
> Marnie
>
>
> On 5/24/07, Martin Ritchie <ri...@apache.org> wrote:
> >
> > I've been trying to tidy up the Java side also. With some recent
> > functional additions but mainly it has been fixing the odd bug. There
> > are a couple of additions that should be JIRAed that would be good to
> > go in with M2 such as the Mina upgrade(QPID-92) along with limiting
> > the size of our send/receive buffers so we reduce the OOM issues with
> > mina, and QPID-465 for JMS compliance.
> >
> > We also have a bug in the client where FailoverExceptions can 'escape'
> > as not all client api calls correctly handle failover,QPID-402.
> >
> > The other serious issue is an un-JIRAed item about our implementation
> > of AMQMessage on the server side. When used in Pub Sub or delivered to
> > multiple queues then looks like we have a problem.
> >
> > Fixing those along with my current work load will probably take us to
> mid
> > June.
> >
> > On 23/05/07, Gordon Sim <gs...@redhat.com> wrote:
> > > Rajith Attapattu wrote:
> > > > Alan can u comment on the c++ side. If I am not mistaken Alan havwe
> > > > already
> > > > identified JIRA's for M2 and is in the process of fixing them.
> > >
> > > Alan is on holiday this week. However we have a fix to the apr_pool
> > > related concurrency issue, channel.flow is implemented and the interop
> > > tests are written. That more or less covers the open JIRAs I think...
> a
> > > couple of them are still marked open as I haven't merged back to trunk
> > > yet (due to other stuff I've got on the go on trunk!).
> > >
> >
> >
> > --
> > Martin Ritchie
> >
>
Re: M2 ?
Posted by Marnie McCormack <ma...@googlemail.com>.
I guess the more 'how long is a piece of string' like question is around
interop ? (Sorry) I'm a bit detached from progress that Rupert's been making
on that front ?
Also, there's the python and ruby clients. On account of the votes (and my
ignorance of the binding process with components in votes for releases) we
are currently compelled to include them. I guess that means they need to
work well with both brokers, or we vote to reduce the scope of the release,
as well as be doc'd enough for users ?
Bfn,
Regards,
Marnie
On 5/24/07, Martin Ritchie <ri...@apache.org> wrote:
>
> I've been trying to tidy up the Java side also. With some recent
> functional additions but mainly it has been fixing the odd bug. There
> are a couple of additions that should be JIRAed that would be good to
> go in with M2 such as the Mina upgrade(QPID-92) along with limiting
> the size of our send/receive buffers so we reduce the OOM issues with
> mina, and QPID-465 for JMS compliance.
>
> We also have a bug in the client where FailoverExceptions can 'escape'
> as not all client api calls correctly handle failover,QPID-402.
>
> The other serious issue is an un-JIRAed item about our implementation
> of AMQMessage on the server side. When used in Pub Sub or delivered to
> multiple queues then looks like we have a problem.
>
> Fixing those along with my current work load will probably take us to mid
> June.
>
> On 23/05/07, Gordon Sim <gs...@redhat.com> wrote:
> > Rajith Attapattu wrote:
> > > Alan can u comment on the c++ side. If I am not mistaken Alan havwe
> > > already
> > > identified JIRA's for M2 and is in the process of fixing them.
> >
> > Alan is on holiday this week. However we have a fix to the apr_pool
> > related concurrency issue, channel.flow is implemented and the interop
> > tests are written. That more or less covers the open JIRAs I think... a
> > couple of them are still marked open as I haven't merged back to trunk
> > yet (due to other stuff I've got on the go on trunk!).
> >
>
>
> --
> Martin Ritchie
>
Re: M2 ?
Posted by Martin Ritchie <ri...@apache.org>.
I've been trying to tidy up the Java side also. With some recent
functional additions but mainly it has been fixing the odd bug. There
are a couple of additions that should be JIRAed that would be good to
go in with M2 such as the Mina upgrade(QPID-92) along with limiting
the size of our send/receive buffers so we reduce the OOM issues with
mina, and QPID-465 for JMS compliance.
We also have a bug in the client where FailoverExceptions can 'escape'
as not all client api calls correctly handle failover,QPID-402.
The other serious issue is an un-JIRAed item about our implementation
of AMQMessage on the server side. When used in Pub Sub or delivered to
multiple queues then looks like we have a problem.
Fixing those along with my current work load will probably take us to mid June.
On 23/05/07, Gordon Sim <gs...@redhat.com> wrote:
> Rajith Attapattu wrote:
> > Alan can u comment on the c++ side. If I am not mistaken Alan havwe
> > already
> > identified JIRA's for M2 and is in the process of fixing them.
>
> Alan is on holiday this week. However we have a fix to the apr_pool
> related concurrency issue, channel.flow is implemented and the interop
> tests are written. That more or less covers the open JIRAs I think... a
> couple of them are still marked open as I haven't merged back to trunk
> yet (due to other stuff I've got on the go on trunk!).
>
--
Martin Ritchie
Re: M2 ?
Posted by Gordon Sim <gs...@redhat.com>.
Rajith Attapattu wrote:
> Alan can u comment on the c++ side. If I am not mistaken Alan havwe
> already
> identified JIRA's for M2 and is in the process of fixing them.
Alan is on holiday this week. However we have a fix to the apr_pool
related concurrency issue, channel.flow is implemented and the interop
tests are written. That more or less covers the open JIRAs I think... a
couple of them are still marked open as I haven't merged back to trunk
yet (due to other stuff I've got on the go on trunk!).
Re: M2 ?
Posted by Rajith Attapattu <ra...@gmail.com>.
Alan can u comment on the c++ side. If I am not mistaken Alan havwe already
identified JIRA's for M2 and is in the process of fixing them.
Rafi how about python stuff?
I have been a bit slow pushing for the M2 release, but perhaps if all stake
holders are ready we up the pace a bit here :)
Regards,
Rajith
On 5/23/07, Rupert Smith <ru...@googlemail.com> wrote:
>
> Or rather, making you think there is more going on than there really is ;)
>
> On 23/05/07, Rupert Smith <ru...@googlemail.com> wrote:
> >
> > Most commits I've done on M2 java recently have been for testing, or
> > adding to the Javadoc, perhaps making it look like there's more going on
> > than you think. Fixing a small bug at the moment, but I'd say the
> > development rate for M2 java is approaching stand still. Unless anyone
> has a
> > different opinion?
> >
> > .Net, C++?
> >
> > Rupert
> >
> > On 22/05/07, Carl Trieloff <cc...@redhat.com> wrote:
> > >
> > >
> > > I see that we still have a reasonable commit rate on the M2 branch. It
> > > would be good to
> > > get a estimate from the group when we want to close it done and cut
> the
> > > release. There
> > > is no rush, just wondering where everyone thinks we are at on
> completing
> > > M2.
> > >
> > > Carl.
> > >
> >
> >
>
Re: M2 ?
Posted by Rupert Smith <ru...@googlemail.com>.
Or rather, making you think there is more going on than there really is ;)
On 23/05/07, Rupert Smith <ru...@googlemail.com> wrote:
>
> Most commits I've done on M2 java recently have been for testing, or
> adding to the Javadoc, perhaps making it look like there's more going on
> than you think. Fixing a small bug at the moment, but I'd say the
> development rate for M2 java is approaching stand still. Unless anyone has a
> different opinion?
>
> .Net, C++?
>
> Rupert
>
> On 22/05/07, Carl Trieloff <cc...@redhat.com> wrote:
> >
> >
> > I see that we still have a reasonable commit rate on the M2 branch. It
> > would be good to
> > get a estimate from the group when we want to close it done and cut the
> > release. There
> > is no rush, just wondering where everyone thinks we are at on completing
> > M2.
> >
> > Carl.
> >
>
>
Re: M2 ?
Posted by Carl Trieloff <cc...@redhat.com>.
Tomas,
As it is a few weeks we can work the languages as they close down, the
review
process will most likely take a few weeks anyway, so if we do .NET last
this may
work out.
Carl.
Tomas Restrepo wrote:
> Carl,
>
>
>> how do you expect the .NET side will need?
>>
>
> It depends what the overall target for M2 is... I know that at least on the
> .NET side, I'd like to get a chance to add the following:
>
> - Basic tuning of the public API
> - Implement Prefetch
> - Make send/receive buffer sizes configurable
> - Make connection timeout configurable
> - Finish tuning the build scripts
> - More tests
>
> I think we'd likely need at least a couple of weeks for that.
>
> Tomas Restrepo
> http://www.winterdom.com/weblog/
>
>
>
>
>
RE: M2 ?
Posted by Tomas Restrepo <to...@devdeo.com>.
Carl,
> how do you expect the .NET side will need?
It depends what the overall target for M2 is... I know that at least on the
.NET side, I'd like to get a chance to add the following:
- Basic tuning of the public API
- Implement Prefetch
- Make send/receive buffer sizes configurable
- Make connection timeout configurable
- Finish tuning the build scripts
- More tests
I think we'd likely need at least a couple of weeks for that.
Tomas Restrepo
http://www.winterdom.com/weblog/
Re: M2 ?
Posted by Carl Trieloff <cc...@redhat.com>.
Tomas,
how do you expect the .NET side will need?
Carl.
Tomas Restrepo wrote:
> On the .NET Client we've been doing significant changes; adding some missing
> functionality, fixing a bunch of race conditions and failures and improving
> the build quality. Still quite a bit of work to go, but at least I'd like to
> try to clean up the public API a bit and add more tests before closing it
> down.
>
>
> Tomas Restrepo
> http://www.winterdom.com/weblog/
>
>
>
>
>> On 22/05/07, Carl Trieloff <cc...@redhat.com> wrote:
>>
>>> I see that we still have a reasonable commit rate on the M2 branch.
>>>
>> It
>>
>>> would be good to
>>> get a estimate from the group when we want to close it done and cut
>>>
>> the
>>
>>> release. There
>>> is no rush, just wondering where everyone thinks we are at on
>>>
>> completing
>>
>>> M2.
>>>
>>> Carl.
>>>
>>>
>
>
RE: M2 ?
Posted by Tomas Restrepo <to...@devdeo.com>.
On the .NET Client we've been doing significant changes; adding some missing
functionality, fixing a bunch of race conditions and failures and improving
the build quality. Still quite a bit of work to go, but at least I'd like to
try to clean up the public API a bit and add more tests before closing it
down.
Tomas Restrepo
http://www.winterdom.com/weblog/
>
> On 22/05/07, Carl Trieloff <cc...@redhat.com> wrote:
> >
> >
> > I see that we still have a reasonable commit rate on the M2 branch.
> It
> > would be good to
> > get a estimate from the group when we want to close it done and cut
> the
> > release. There
> > is no rush, just wondering where everyone thinks we are at on
> completing
> > M2.
> >
> > Carl.
> >
Re: M2 ?
Posted by Rupert Smith <ru...@googlemail.com>.
Most commits I've done on M2 java recently have been for testing, or adding
to the Javadoc, perhaps making it look like there's more going on than you
think. Fixing a small bug at the moment, but I'd say the development rate
for M2 java is approaching stand still. Unless anyone has a different
opinion?
.Net, C++?
Rupert
On 22/05/07, Carl Trieloff <cc...@redhat.com> wrote:
>
>
> I see that we still have a reasonable commit rate on the M2 branch. It
> would be good to
> get a estimate from the group when we want to close it done and cut the
> release. There
> is no rush, just wondering where everyone thinks we are at on completing
> M2.
>
> Carl.
>
M2 ?
Posted by Carl Trieloff <cc...@redhat.com>.
I see that we still have a reasonable commit rate on the M2 branch. It
would be good to
get a estimate from the group when we want to close it done and cut the
release. There
is no rush, just wondering where everyone thinks we are at on completing M2.
Carl.
Re: C++ broker issues?
Posted by Kevin Smith <ks...@redhat.com>.
Martin Ritchie wrote:
> I'm confused. Kevin do you not have permission to commit the patch
> yourself? If your account isn't working correctly then we should take
> a look at that, we did vote to give you commit rights after all.
>
> On 22/05/07, Gordon Sim <gs...@redhat.com> wrote:
>> Andrew Stitcher wrote:
>> > Please do apply Kevin's patch. I'm getting ridiculously bogged down
>> > doing the pthreads work. So at least apply the patch and then do the
>> > performance tests, if the performance isn't too bad then it'd be easier
>> > not to merge my changes.
>>
>> Done (r540511 on M2). Seems to fix the problems I was seeing nicely.
>> Thanks again Kevin!
>>
>
>
I do have access but I am uncomfortable committing changes to the C++ broker
without peer review. It's been literally years since I coded anything of
significance in C++...
--Kevin
Re: C++ broker issues?
Posted by Martin Ritchie <ri...@apache.org>.
I'm confused. Kevin do you not have permission to commit the patch
yourself? If your account isn't working correctly then we should take
a look at that, we did vote to give you commit rights after all.
On 22/05/07, Gordon Sim <gs...@redhat.com> wrote:
> Andrew Stitcher wrote:
> > Please do apply Kevin's patch. I'm getting ridiculously bogged down
> > doing the pthreads work. So at least apply the patch and then do the
> > performance tests, if the performance isn't too bad then it'd be easier
> > not to merge my changes.
>
> Done (r540511 on M2). Seems to fix the problems I was seeing nicely.
> Thanks again Kevin!
>
--
Martin Ritchie
Re: C++ broker issues?
Posted by Gordon Sim <gs...@redhat.com>.
Andrew Stitcher wrote:
> Please do apply Kevin's patch. I'm getting ridiculously bogged down
> doing the pthreads work. So at least apply the patch and then do the
> performance tests, if the performance isn't too bad then it'd be easier
> not to merge my changes.
Done (r540511 on M2). Seems to fix the problems I was seeing nicely.
Thanks again Kevin!
Re: C++ broker issues?
Posted by Andrew Stitcher <as...@redhat.com>.
On Tue, 2007-05-22 at 09:12 +0100, Gordon Sim wrote:
> Robert Godfrey wrote:
> > OK - just checking... I know you C++ guys are quite a way ahead on
> > trunk of where we are on M2... And this is the sort of thing that
> > really needs fixing before we release to a wider audience...
>
> Agreed; we won't release the c++ side until this is fixed.
>
> > Any idea of an ETA for the fix - I'm quite keen to start running our
> > performance tests again...
>
> Andrew can give the most accurate ETA. I'm happy to apply Kevin's patch
> to M2 in the meantime - I don't think it will result in much conflict
> with the removal of apr based threads/mutexes etc., some of the patched
> classes may be removed at a later stage is all.
Please do apply Kevin's patch. I'm getting ridiculously bogged down
doing the pthreads work. So at least apply the patch and then do the
performance tests, if the performance isn't too bad then it'd be easier
not to merge my changes.
Andrew
Re: C++ broker issues?
Posted by Gordon Sim <gs...@redhat.com>.
Robert Godfrey wrote:
> OK - just checking... I know you C++ guys are quite a way ahead on
> trunk of where we are on M2... And this is the sort of thing that
> really needs fixing before we release to a wider audience...
Agreed; we won't release the c++ side until this is fixed.
> Any idea of an ETA for the fix - I'm quite keen to start running our
> performance tests again...
Andrew can give the most accurate ETA. I'm happy to apply Kevin's patch
to M2 in the meantime - I don't think it will result in much conflict
with the removal of apr based threads/mutexes etc., some of the patched
classes may be removed at a later stage is all.
Re: C++ broker issues?
Posted by Robert Godfrey <ro...@gmail.com>.
OK - just checking... I know you C++ guys are quite a way ahead on
trunk of where we are on M2... And this is the sort of thing that
really needs fixing before we release to a wider audience...
Any idea of an ETA for the fix - I'm quite keen to start running our
performance tests again...
Cheers,
Rob
On 22/05/07, Gordon Sim <gs...@redhat.com> wrote:
> Robert Godfrey wrote:
> > Are the changes that Andrew is working on aimed at the M2 release?
>
> They were, yes. (Though I believe he's making them to trunk, then
> merging). Kevin's patch will be a good backup though. It was one of the
> approaches considered (and actually the approach used when the apr
> classes were originally created).
>
Re: C++ broker issues?
Posted by Gordon Sim <gs...@redhat.com>.
Robert Godfrey wrote:
> Are the changes that Andrew is working on aimed at the M2 release?
They were, yes. (Though I believe he's making them to trunk, then
merging). Kevin's patch will be a good backup though. It was one of the
approaches considered (and actually the approach used when the apr
classes were originally created).
Re: C++ broker issues?
Posted by Robert Godfrey <ro...@gmail.com>.
Are the changes that Andrew is working on aimed at the M2 release?
Cheers,
Rob
On 22/05/07, Gordon Sim <gs...@redhat.com> wrote:
> Kevin Smith wrote:
> > Attached is my attempt at fixing this problem. There are, no doubt,
> > better ways to do this. This is just the way I hacked to fix the
> > fall-over-with-three-clients problem I was having.
>
> Thanks, Kevin!
>
> > My patch does three things:
> >
> > 1) Modifies the APRPool class to use alloc/free semantics for APR memory
> > pools. Each time a caller calls APRPool::get() they'll their own pool
> > reference. I've fixed up all the call sites I can find to also call
> > APRPool::free() at the appropriate time.
>
> Yes, thats what I did originally as well.
>
> > 2) Caches freed APR memory pools in a STL stack. This cuts down on the
> > number of memory pools created overall.
>
> Nice idea.
>
> > 3) As a result of doing #1 and #2 I've introduced a guard mutex around
> > APRPool::get() and APRPool::free(). This is to prevent concurrent access
> > to the memory pool cache. If it's too heavyweight, the mutex along with
> > the caching mechanism could be removed entirely.
> >
> > Like I said before, my C++ is very rusty so I make no guarantees about
> > code correctness or quality. I've fixed the leaks I could find (and
> > understand) from valgrind so I _think_ this isn't any leakier than the
> > code was before my patch.
> >
> > I've run a mini-stress test (10 concurrent clients sending messages via
> > a topic exchange) multiple times and haven't been able to get the broker
> > to fall over. Memory usage seems pretty stable once the broker is up and
> > all clients have connected.
>
> Thanks again for sending in the patch. Andrew is working on replacing
> apr mutexes etc with pthreads. However this change might still be a
> useful complement to that covering some of the other apr objects?
>
Re: C++ broker issues?
Posted by Gordon Sim <gs...@redhat.com>.
Kevin Smith wrote:
> Attached is my attempt at fixing this problem. There are, no doubt,
> better ways to do this. This is just the way I hacked to fix the
> fall-over-with-three-clients problem I was having.
Thanks, Kevin!
> My patch does three things:
>
> 1) Modifies the APRPool class to use alloc/free semantics for APR memory
> pools. Each time a caller calls APRPool::get() they'll their own pool
> reference. I've fixed up all the call sites I can find to also call
> APRPool::free() at the appropriate time.
Yes, thats what I did originally as well.
> 2) Caches freed APR memory pools in a STL stack. This cuts down on the
> number of memory pools created overall.
Nice idea.
> 3) As a result of doing #1 and #2 I've introduced a guard mutex around
> APRPool::get() and APRPool::free(). This is to prevent concurrent access
> to the memory pool cache. If it's too heavyweight, the mutex along with
> the caching mechanism could be removed entirely.
>
> Like I said before, my C++ is very rusty so I make no guarantees about
> code correctness or quality. I've fixed the leaks I could find (and
> understand) from valgrind so I _think_ this isn't any leakier than the
> code was before my patch.
>
> I've run a mini-stress test (10 concurrent clients sending messages via
> a topic exchange) multiple times and haven't been able to get the broker
> to fall over. Memory usage seems pretty stable once the broker is up and
> all clients have connected.
Thanks again for sending in the patch. Andrew is working on replacing
apr mutexes etc with pthreads. However this change might still be a
useful complement to that covering some of the other apr objects?
Re: C++ broker issues?
Posted by Kevin Smith <ks...@redhat.com>.
Kevin Smith wrote:
> Kevin Smith wrote:
>> Gordon Sim wrote:
>>> Kevin Smith wrote:
>>>> Gordon Sim wrote:
>>>>> Kevin Smith wrote:
>>>>>> Thanks for the update! If there are any patches for this against
>>>>>> M2, I'd be willing to help test.
>>>>>
>>>>> Fantastic! We are aiming to fix the M2 branch first and if you
>>>>> could re-run your test at that time to ensure the problem is fixed
>>>>> that would be much appreciated.
>>>>>
>>>> I've hacked together a fix against M2 which seems to address the
>>>> problems I was seeing. I could post my patch to the list depending
>>>> on how soon a fix is expected against M2. My C++ is pretty rusty, so
>>>> I make no guarantees about code correctness or quality :)
>>>
>>> Yes, I'd be interested in seeing the patch since you have it anyway!
>>>
>> And I spoke waaaay to soon :-)
>>
>> My C++ is rustier than I originally thought given the sorry report I
>> just got from valgrind. I'll post something as soon as I get it to
>> quit leaking like a sieve...
>>
>> --Kevin
>>
> Attached is my attempt at fixing this problem. There are, no doubt,
> better ways to do this. This is just the way I hacked to fix the
> fall-over-with-three-clients problem I was having.
>
> My patch does three things:
>
> 1) Modifies the APRPool class to use alloc/free semantics for APR memory
> pools. Each time a caller calls APRPool::get() they'll their own pool
> reference. I've fixed up all the call sites I can find to also call
> APRPool::free() at the appropriate time.
>
> 2) Caches freed APR memory pools in a STL stack. This cuts down on the
> number of memory pools created overall.
>
> 3) As a result of doing #1 and #2 I've introduced a guard mutex around
> APRPool::get() and APRPool::free(). This is to prevent concurrent access
> to the memory pool cache. If it's too heavyweight, the mutex along with
> the caching mechanism could be removed entirely.
>
> Like I said before, my C++ is very rusty so I make no guarantees about
> code correctness or quality. I've fixed the leaks I could find (and
> understand) from valgrind so I _think_ this isn't any leakier than the
> code was before my patch.
>
> I've run a mini-stress test (10 concurrent clients sending messages via
> a topic exchange) multiple times and haven't been able to get the broker
> to fall over. Memory usage seems pretty stable once the broker is up and
> all clients have connected.
>
>
> --Kevin
Let's try this again. I think my previous email got stuck in moderation due to
the size of the patch. I'm attaching it again as a tarball, maybe this will make
it thru.
Re: C++ broker issues?
Posted by Kevin Smith <ks...@redhat.com>.
Kevin Smith wrote:
> Gordon Sim wrote:
>> Kevin Smith wrote:
>>> Gordon Sim wrote:
>>>> Kevin Smith wrote:
>>>>> Thanks for the update! If there are any patches for this against
>>>>> M2, I'd be willing to help test.
>>>>
>>>> Fantastic! We are aiming to fix the M2 branch first and if you could
>>>> re-run your test at that time to ensure the problem is fixed that
>>>> would be much appreciated.
>>>>
>>> I've hacked together a fix against M2 which seems to address the
>>> problems I was seeing. I could post my patch to the list depending on
>>> how soon a fix is expected against M2. My C++ is pretty rusty, so I
>>> make no guarantees about code correctness or quality :)
>>
>> Yes, I'd be interested in seeing the patch since you have it anyway!
>>
> And I spoke waaaay to soon :-)
>
> My C++ is rustier than I originally thought given the sorry report I
> just got from valgrind. I'll post something as soon as I get it to quit
> leaking like a sieve...
>
> --Kevin
>
Attached is my attempt at fixing this problem. There are, no doubt, better ways
to do this. This is just the way I hacked to fix the
fall-over-with-three-clients problem I was having.
My patch does three things:
1) Modifies the APRPool class to use alloc/free semantics for APR memory pools.
Each time a caller calls APRPool::get() they'll their own pool reference. I've
fixed up all the call sites I can find to also call APRPool::free() at the
appropriate time.
2) Caches freed APR memory pools in a STL stack. This cuts down on the number of
memory pools created overall.
3) As a result of doing #1 and #2 I've introduced a guard mutex around
APRPool::get() and APRPool::free(). This is to prevent concurrent access to the
memory pool cache. If it's too heavyweight, the mutex along with the caching
mechanism could be removed entirely.
Like I said before, my C++ is very rusty so I make no guarantees about code
correctness or quality. I've fixed the leaks I could find (and understand) from
valgrind so I _think_ this isn't any leakier than the code was before my patch.
I've run a mini-stress test (10 concurrent clients sending messages via a topic
exchange) multiple times and haven't been able to get the broker to fall over.
Memory usage seems pretty stable once the broker is up and all clients have
connected.
--Kevin
Re: C++ broker issues?
Posted by Kevin Smith <ks...@redhat.com>.
Gordon Sim wrote:
> Kevin Smith wrote:
>> Gordon Sim wrote:
>>> Kevin Smith wrote:
>>>> Thanks for the update! If there are any patches for this against M2,
>>>> I'd be willing to help test.
>>>
>>> Fantastic! We are aiming to fix the M2 branch first and if you could
>>> re-run your test at that time to ensure the problem is fixed that
>>> would be much appreciated.
>>>
>> I've hacked together a fix against M2 which seems to address the
>> problems I was seeing. I could post my patch to the list depending on
>> how soon a fix is expected against M2. My C++ is pretty rusty, so I
>> make no guarantees about code correctness or quality :)
>
> Yes, I'd be interested in seeing the patch since you have it anyway!
>
And I spoke waaaay to soon :-)
My C++ is rustier than I originally thought given the sorry report I just got
from valgrind. I'll post something as soon as I get it to quit leaking like a
sieve...
--Kevin
Re: C++ broker issues?
Posted by Gordon Sim <gs...@redhat.com>.
Kevin Smith wrote:
> Gordon Sim wrote:
>> Kevin Smith wrote:
>>> Thanks for the update! If there are any patches for this against M2,
>>> I'd be willing to help test.
>>
>> Fantastic! We are aiming to fix the M2 branch first and if you could
>> re-run your test at that time to ensure the problem is fixed that
>> would be much appreciated.
>>
> I've hacked together a fix against M2 which seems to address the
> problems I was seeing. I could post my patch to the list depending on
> how soon a fix is expected against M2. My C++ is pretty rusty, so I make
> no guarantees about code correctness or quality :)
Yes, I'd be interested in seeing the patch since you have it anyway!
Re: C++ broker issues?
Posted by Kevin Smith <ks...@redhat.com>.
Gordon Sim wrote:
> Kevin Smith wrote:
>> Thanks for the update! If there are any patches for this against M2,
>> I'd be willing to help test.
>
> Fantastic! We are aiming to fix the M2 branch first and if you could
> re-run your test at that time to ensure the problem is fixed that would
> be much appreciated.
>
I've hacked together a fix against M2 which seems to address the problems I was
seeing. I could post my patch to the list depending on how soon a fix is
expected against M2. My C++ is pretty rusty, so I make no guarantees about code
correctness or quality :)
--Kevin
Re: C++ broker issues?
Posted by Gordon Sim <gs...@redhat.com>.
Kevin Smith wrote:
> Thanks for the update! If there are any patches for this against M2, I'd
> be willing to help test.
Fantastic! We are aiming to fix the M2 branch first and if you could
re-run your test at that time to ensure the problem is fixed that would
be much appreciated.
Re: C++ broker issues?
Posted by Kevin Smith <ks...@redhat.com>.
Gordon Sim wrote:
> Kevin Smith wrote:
>> I've heard that the C++ broker has concurrency issues -- can anyone
>> confirm this? I'm running into a situation where more than two
>> concurrent clients cause the broker to wedge itself at 100% CPU and
>> never return.
>
> Yes, I'm afraid that is the case. It uses a single apr pool for all apr
> allocated objects which means that any concurrent destruction of such
> objects can lead to an infinite loop in the apr pool cleanup code.
>
> A fix is hopefully coming shortly.
>
Thanks for the update! If there are any patches for this against M2, I'd be
willing to help test.
--Kevin
Re: C++ broker issues?
Posted by Gordon Sim <gs...@redhat.com>.
Kevin Smith wrote:
> I've heard that the C++ broker has concurrency issues -- can anyone
> confirm this? I'm running into a situation where more than two
> concurrent clients cause the broker to wedge itself at 100% CPU and
> never return.
Yes, I'm afraid that is the case. It uses a single apr pool for all apr
allocated objects which means that any concurrent destruction of such
objects can lead to an infinite loop in the apr pool cleanup code.
A fix is hopefully coming shortly.
Re: C++ broker issues?
Posted by Andrew Stitcher <as...@redhat.com>.
I'm still working on the fix, which is (sigh) taking longer than I'd
have hoped.
The fix will actually go into the trunk first then very shortly after
that into M2 - I was already working in the same area in the trunk so it
was easier to do it that way round.
Andrew
On Wed, 2007-05-16 at 13:54 +0100, Rupert Smith wrote:
> Yes, I'd heard from Rob that Gordon might have fixed this? Maybe a checkin
> for the fix is still pending? I wanted to run my performance test suite
> against the C++ broker yesterday, on M2 branch, but found that if I didn't
> limit the message rate severely it would jam up. Would be nice to know the
> status.
>
> Rupert
>
> On 16/05/07, Kevin Smith <ks...@redhat.com> wrote:
> >
> > I've heard that the C++ broker has concurrency issues -- can anyone
> > confirm
> > this? I'm running into a situation where more than two concurrent clients
> > cause
> > the broker to wedge itself at 100% CPU and never return.
> >
> > --Kevin
> >
Re: C++ broker issues?
Posted by Rupert Smith <ru...@googlemail.com>.
Yes, I'd heard from Rob that Gordon might have fixed this? Maybe a checkin
for the fix is still pending? I wanted to run my performance test suite
against the C++ broker yesterday, on M2 branch, but found that if I didn't
limit the message rate severely it would jam up. Would be nice to know the
status.
Rupert
On 16/05/07, Kevin Smith <ks...@redhat.com> wrote:
>
> I've heard that the C++ broker has concurrency issues -- can anyone
> confirm
> this? I'm running into a situation where more than two concurrent clients
> cause
> the broker to wedge itself at 100% CPU and never return.
>
> --Kevin
>