DIRMINA-391 vs DIRMINA-576

[Dan Lawesson <da...@kodgruvan.se>]

Hi,

Until yesterday I was under the impression that heap buffers are the  
way to go and direct buffers are more of a relic of at most historical  
value. This was a belief that was at least partly supported by the  
Apache MINA project documentation of the upcoming version 2.0.  
Yesterday my view on heap buffers was completely reversed when I  
witnessed a perfectly working legacy system (Java 1.4.2 something)  
barf and die due to lack of direct memory even though we did not  
explicitly use any direct heap buffers at all. Post mortem heap dumps  
show that there had been a creation of 100 direct buffers per second  
and none of them were cleaned since heap memory was fine and no full  
GC sweep was triggered. The heap contained a chain of 200k cleaners  
after 30 minutes of uptime. The reason I have not witnessed this  
before is probably that usually these systems also use a lot of heap  
memory and therefore trigger full GC which triggers the cleaners.

After a while we realized that in case you use heap buffers, you get a  
temporary direct buffer. Clearly, at least in this old JVM these  
temporary direct buffers are not even pooled (that was the WTF of the  
day!).

This lead me to DIRMINA-391 http://issues.apache.org/jira/browse/DIRMINA-391 
. Trustin Lee advocates heap buffers, even though the issue presents a  
strong case against them. Does this mean that this behavior of using  
non-pooled temporary direct buffers is a thing of the past, i.e., more  
modern JVMs are smarter than that? This is the only explanation I can  
find to why MINA 2.0 uses heap buffers per default and also has  
dropped the pooling feature of older MINA versions.

The reason I write this mail is that both https://issues.apache.org/jira/browse/DIRMINA-576 
  and the linked Sun issue seem to contradict this. They both make a  
strong case for always using pooled direct buffers. What is the  
community consensus on heap buffers?

Best Regards,
/Dan

Re: DIRMINA-391 vs DIRMINA-576

[Dan Lawesson <da...@kodgruvan.se>]

26 jan 2010 kl. 16.55 skrev Emmanuel LŽcharny:

> Dan Lawesson a écrit :
>> Hi,
>>
>> Until yesterday I was under the impression that heap buffers are  
>> the way to go and direct buffers are more of a relic of at most  
>> historical value. This was a belief that was at least partly  
>> supported by the Apache MINA project documentation of the upcoming  
>> version 2.0. Yesterday my view on heap buffers was completely  
>> reversed when I witnessed a perfectly working legacy system (Java  
>> 1.4.2 something) barf and die due to lack of direct memory even  
>> though we did not explicitly use any direct heap buffers at all.  
>> Post mortem heap dumps show that there had been a creation of 100  
>> direct buffers per second and none of them were cleaned since heap  
>> memory was fine and no full GC sweep was triggered. The heap  
>> contained a chain of 200k cleaners after 30 minutes of uptime. The  
>> reason I have not witnessed this before is probably that usually  
>> these systems also use a lot of heap memory and therefore trigger  
>> full GC which triggers the cleaners.
>>
>> After a while we realized that in case you use heap buffers, you  
>> get a temporary direct buffer. Clearly, at least in this old JVM  
>> these temporary direct buffers are not even pooled (that was the  
>> WTF of the day!).
>>
>> This lead me to DIRMINA-391 http://issues.apache.org/jira/browse/DIRMINA-391 
>> . Trustin Lee advocates heap buffers, even though the issue  
>> presents a strong case against them. Does this mean that this  
>> behavior of using non-pooled temporary direct buffers is a thing of  
>> the past, i.e., more modern JVMs are smarter than that? This is the  
>> only explanation I can find to why MINA 2.0 uses heap buffers per  
>> default and also has dropped the pooling feature of older MINA  
>> versions.
>>
>> The reason I write this mail is that both https://issues.apache.org/jira/browse/DIRMINA-576 
>>  and the linked Sun issue seem to contradict this. They both make a  
>> strong case for always using pooled direct buffers. What is the  
>> community consensus on heap buffers?
> Concensus, I don't know. My strong opinion is that direct buffers  
> are to be used when dealing with low level IO, and heap buffers must  
> be used for everything else.
>
> The fact that you can kill your OS because you eat all of it's  
> memory seems to me a big no-no for DirectBuffer. If I use Heap  
> buffers and if I suck up all my JVM memory, too bad for me, my code  
> and my salary, but at least, it impacts only my application's JVM.
>
> In your case, the Direct buffers are used to exchange data with a  
> socket in a IOread, done by the JVM : makes perfect sense. The  
> question is why aren't they cleaned ?
>
> I have to dig back into those issues I guess...

Thank you very much for your reply. It is now clear that I had not  
researched the issue thoroughly enough before asking the question. The  
JVM does pool the direct buffers, very much to the contrary of my ill- 
informed statement above.

I was lead to think that there was no pooling because there was  
approximately 100 new allocations per second in the system I analyzed.  
For some reason it did not occur to me that the clever and efficient  
way to implement the pooling is to do it in thread local pools, thus  
eliminating any need for synchronization. Comments on the relevant Sun  
bug issues and DIRMINA-576 seemed to further reinforce my suspicion  
that something was actually wrong in the JVM handling of the buffers.  
My mistake was to not understand that the pooling is thread local and  
that I focused my attention on the protocol handling low level module  
of the system, failing to realize that a business logic part of the  
system called write operations on the protocol module in new (as in  
"new Thread()") threads due to a peculiarity of the implementation of  
a timer callback.

Now I believe that I can safely return to my previous belief that heap  
buffers are to be preferred in essentially all situations (in line  
with the statements of Trustin Lee in DIRMINA-391) as long I make sure  
that the total number of threads involved in read() and write() are  
limited (by for example pooling of threads). For the problem at hand  
it was not feasible to redesign the thread model for the business  
logic module, so we settled for explicit (non-thread local) pooling of  
direct buffers used for the write operation.

Re: DIRMINA-391 vs DIRMINA-576

[Emmanuel LŽcharny <el...@gmail.com>]

Dan Lawesson a écrit :
> Hi,
>
> Until yesterday I was under the impression that heap buffers are the 
> way to go and direct buffers are more of a relic of at most historical 
> value. This was a belief that was at least partly supported by the 
> Apache MINA project documentation of the upcoming version 2.0. 
> Yesterday my view on heap buffers was completely reversed when I 
> witnessed a perfectly working legacy system (Java 1.4.2 something) 
> barf and die due to lack of direct memory even though we did not 
> explicitly use any direct heap buffers at all. Post mortem heap dumps 
> show that there had been a creation of 100 direct buffers per second 
> and none of them were cleaned since heap memory was fine and no full 
> GC sweep was triggered. The heap contained a chain of 200k cleaners 
> after 30 minutes of uptime. The reason I have not witnessed this 
> before is probably that usually these systems also use a lot of heap 
> memory and therefore trigger full GC which triggers the cleaners.
>
> After a while we realized that in case you use heap buffers, you get a 
> temporary direct buffer. Clearly, at least in this old JVM these 
> temporary direct buffers are not even pooled (that was the WTF of the 
> day!).
>
> This lead me to DIRMINA-391 
> http://issues.apache.org/jira/browse/DIRMINA-391. Trustin Lee 
> advocates heap buffers, even though the issue presents a strong case 
> against them. Does this mean that this behavior of using non-pooled 
> temporary direct buffers is a thing of the past, i.e., more modern 
> JVMs are smarter than that? This is the only explanation I can find to 
> why MINA 2.0 uses heap buffers per default and also has dropped the 
> pooling feature of older MINA versions.
>
> The reason I write this mail is that both 
> https://issues.apache.org/jira/browse/DIRMINA-576 and the linked Sun 
> issue seem to contradict this. They both make a strong case for always 
> using pooled direct buffers. What is the community consensus on heap 
> buffers?
Concensus, I don't know. My strong opinion is that direct buffers are to 
be used when dealing with low level IO, and heap buffers must be used 
for everything else.

The fact that you can kill your OS because you eat all of it's memory 
seems to me a big no-no for DirectBuffer. If I use Heap buffers and if I 
suck up all my JVM memory, too bad for me, my code and my salary, but at 
least, it impacts only my application's JVM.

In your case, the Direct buffers are used to exchange data with a socket 
in a IOread, done by the JVM : makes perfect sense. The question is why 
aren't they cleaned ?

I have to dig back into those issues I guess...