You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by nigro_franz <ni...@gmail.com> on 2017/03/06 09:10:17 UTC

[DISCUSS] OutOfDirectMemory on high memory pressure 4 NettyConnection

Hi guys!

I'm writing here to have your opinion about what could look like an "issue"
that I've found in the NettyConnection (InVM suffer of the same issue, but
cover different use cases) of Artemis.
If you push enough pressure on the NettyConnection::createTransportBuffer
with huge messages and/or an high count of living connections there will be
a moment in which Artemis will throw an OutOfDirectMemory exception due to
the reached limit imposed by the io.netty.maxDirectMemory property.
I've noticed that the most there will be the speed difference between the
journalling phase and the messages arrival (through the connection) the most
it will be the case that the exception will be thrown, but there are
multiple scenarios that could make it happens.

IMHO this issue is not something that could be solved switching to an
unpooled Heap ByteBuf allocator: it will shift the responsability to the
JVM's GC, leading to even worst case effects, like infinite and
unpredictable GC pauses (eg due to faster aging of garbage and/or homougous
space exaustions depending by the rate/size/fragmentation of the byte[]
allocations).

What i'm seeing is more a design needs: an effective way to backpressure the
different process layers of the messages.
Netty doesn't provide it if not in the form of that exception, but for an
application like Artemis I think it need to be done something that can be
queried, monitored and tuned.

Having an effective way to monitor and bound the latency between each stage
will enable the lower stages (ie in front of the I/O) to choose a proper
strategy, like dropping new connections when overloaded or blocking them
until proper timeouts/freeing of resources.

I've noticed that all the concurrent stages of message processing are backed
by unbounded queues (ie LinkedBlockingQueue or ConcurrentLinkedQueue) hidden
by the Executors/ExecutorServices and that's the point that need to be
improved: switching to bounded queues and real message passing (not
Runnable) between the stages will let the application to know in a simple
and effective way when the next stages can't keepup with the processing
requests.

Sorry for this long post, but I've spent a lot of time figuring out (and
trying/testing) different approaches before arriving to this conclusion...

what do you think?

Cheers,
Franz



--
View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-OutOfDirectMemory-on-high-memory-pressure-4-NettyConnection-tp4722964.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: [DISCUSS] OutOfDirectMemory on high memory pressure 4 NettyConnection

Posted by nigro_franz <ni...@gmail.com>.
>This one is tricky... The Linux kernel has some thread controls... so 
>the writing has to be done through a single thread... context switch 
>would make some extra IO on disks and performance would suffer... 

Yes, you're right,sorry I've missed some context :P
I mean a configuration like this:

clients that ned to append to the journal  --[APPEND REQUESTS QUEUE]--> 
journal appender (poll the requests) that uses the SequentialFile
implementation

The Journal appender will be single threaded and would use the chosen
SequentialFile (as it is) implementation, draining from the append requests
queue the requests to write into the journal and batching when necessary if
it will help to improve the performance (coalescing the flushes).
The same could be done on the InVM/NettyConnection too: the thread that is
responsible to batch the writes to the sockets could receive its requests
from a queue without any further need to allocate new buffers every time...
My other concern here is how Netty could behave...



--
View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-OutOfDirectMemory-on-high-memory-pressure-4-NettyConnection-tp4722964p4723157.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: [DISCUSS] OutOfDirectMemory on high memory pressure 4 NettyConnection

Posted by Clebert Suconic <cl...@gmail.com>.
> 2) the management of the journal (the write operations in particular)
> makes sense?

This one is tricky... The Linux kernel has some thread controls... so
the writing has to be done through a single thread... context switch
would make some extra IO on disks and performance would suffer...


we may eventually need to use two threads.. but bind them to IO...
this is hard core...  even for NIO (not just libaio). I bet it would
make some difference on mapped file as well.

Re: [DISCUSS] OutOfDirectMemory on high memory pressure 4 NettyConnection

Posted by nigro_franz <ni...@gmail.com>.
Thanks for the feedback Clebert!

>I think we should use some sort of pool on messages. Reusing the big 
>pools before the messages are released. 
It could work!
Just thinking loud; in order to implement this pool efficiently my advice is
to not use a solution similar to the Netty pool, in which the object
lifecycle could span between threads: Netty itself shows that it doesn't
scale well for high throughput scenarios and risks to be error prone on
leaking messages (it needs to implement a leak detector etc etc).
I've already built some experiments around this approach if anyone is
interested in the results, but could worth try it too!

Instead I'm proposing another form of pooling, that will provide the same
effect of recycling memory but as a consequence and not as a mean: bounded
queues between the message process stages.
Having bounded concurrent queues will provide the pooling for free and a
mean to measure the utilization between all the concurrent stages (eg
incoming,encoding, journaling ,decoding, outgoing etc etc) recognizing when
the system is overloaded in any of its part and letting the dev/user to tune
it consequently.

>One of the changes as part of my big PR was to not duplicate buffers 
>on every deliver.. that alone should release some pressure on GC for 
>cases where the message is delivered to more than one destination 
>(topic subscriptions). 
And it is a super plus to improve the throughput for sure (I've already
benchmarked it, delivering outstanding improvements indeed ): bound the
memory and the max latency is only an additional tool to exploit further
this improved throughput, making it sustainable.

>what would you change here? what would you use for such things? 
I would bound first (putting a bounded queue in front of) the stages that
could be slowed down by external factors (like I/O speeds), that need to
inform the previous ones of being overloaded and that could benefit by any
form a batching:
1) the incoming/outcoming data phase of the connections (InVM or Netty)
2) the management of the journal (the write operations in particular)
makes sense?






--
View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-OutOfDirectMemory-on-high-memory-pressure-4-NettyConnection-tp4722964p4722999.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: [DISCUSS] OutOfDirectMemory on high memory pressure 4 NettyConnection

Posted by Clebert Suconic <cl...@gmail.com>.
On Mon, Mar 6, 2017 at 4:10 AM, nigro_franz <ni...@gmail.com> wrote:
> Hi guys!
>
> I'm writing here to have your opinion about what could look like an "issue"
> that I've found in the NettyConnection (InVM suffer of the same issue, but
> cover different use cases) of Artemis.
> If you push enough pressure on the NettyConnection::createTransportBuffer
> with huge messages and/or an high count of living connections there will be
> a moment in which Artemis will throw an OutOfDirectMemory exception due to
> the reached limit imposed by the io.netty.maxDirectMemory property.
> I've noticed that the most there will be the speed difference between the
> journalling phase and the messages arrival (through the connection) the most
> it will be the case that the exception will be thrown, but there are
> multiple scenarios that could make it happens.
>
> IMHO this issue is not something that could be solved switching to an
> unpooled Heap ByteBuf allocator: it will shift the responsability to the
> JVM's GC, leading to even worst case effects, like infinite and
> unpredictable GC pauses (eg due to faster aging of garbage and/or homougous
> space exaustions depending by the rate/size/fragmentation of the byte[]
> allocations).
>
> What i'm seeing is more a design needs: an effective way to backpressure the
> different process layers of the messages.
> Netty doesn't provide it if not in the form of that exception, but for an
> application like Artemis I think it need to be done something that can be
> queried, monitored and tuned.
>
> Having an effective way to monitor and bound the latency between each stage
> will enable the lower stages (ie in front of the I/O) to choose a proper
> strategy, like dropping new connections when overloaded or blocking them
> until proper timeouts/freeing of resources.


I think we should use some sort of pool on messages. Reusing the big
pools before the messages are released.

One of the changes as part of my big PR was to not duplicate buffers
on every deliver.. that alone should release some pressure on GC for
cases where the message is delivered to more than one destination
(topic subscriptions).

That should also make it possible to optimize countings.



>
> I've noticed that all the concurrent stages of message processing are backed
> by unbounded queues (ie LinkedBlockingQueue or ConcurrentLinkedQueue) hidden
> by the Executors/ExecutorServices and that's the point that need to be
> improved: switching to bounded queues and real message passing (not
> Runnable) between the stages will let the application to know in a simple
> and effective way when the next stages can't keepup with the processing
> requests.

what would you change here? what would you use for such things?