You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Eric Charles <er...@u-mangate.com> on 2010/04/24 14:13:18 UTC
OOM with Netty
Hi All,
I just got oom with netty impl.
See http://apache.u-mangate.com/james/oom/20090424.png
(http://apache.u-mangate.com/james/oom/index.html for history).
Everything was fine with the trunk of last week that was really stable.
I redeployed 3 days ago a new trunk version after a stop/start that
showed ClassCastException of ImapMessage (ClassCastException was solved
with the redeploy)
Short infos about the today oom:
- The logs had IOException "Bad file descriptor" (see copy of the
exception after this mail)
- The begining time of these exceptions in the log is 00:10, and there
are numerous ones during 1 minute.
- The heap dump automatically generated via
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/ occured on 00:10,
exactly on the same time the IOException "Bad file descriptor" began.
- I forgot to lsof to see the open files... but when if the error came
from too many open files, the received exception in the log would have
been "java.io.IOException: too many open files".
- Seems like an infinite loop as Norman anticipated, but now stacktraces
from logs seem to confirm this.
- Looking at line 150 of
http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=markup
, out.write(line) seem to be the place that triggers the exception, out
variable being = (OutputStream)
session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
- The server filesystem seems sane (new hardware, running with enough
privilege, other applications running on the same machine without any
problem). I don't see why I have a "Bad file descriptor"
There are file in tmp with creation date that span 1 week (so not
produced on 00:10):
- A few messagestream*.ms
- More rewindable*.tmp
I restarted james.
It still showed some issues (one account via pop3 gave errors in my client).
oouch...
I updated from trunk, redeployed, copied the var directory to migrate
the account/domain/mail database, and restarted.
Hopefully, now everything is working as before.
So, we've got now a useful stacktrace that clearly shows where the oom
"trigger" could be.
I'm pretty sure that jconsole would have shown the same behaviour as
http://apache.u-mangate.com/james/oom/oom.png
Possible cause on the SMTP channel:
- A weird behaviour of the smtp clients (hacking or respectfulness
protocols clients)
- A design in james causing infinite loop: thread clash (a thread
accessing the same file as another thread?,...)
- Additional James config (spamassassin or whatever) that leads to such
behaviour
- Hardware problem
Currently, I run with default config + jdbcdomainlist.
I except to receive another oom one of these days, unless the oom comes
from an unactivated config (spamassassin or whatever).
Tks,
Eric
ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
processing DATA.
java.io.IOException: Bad file descriptor
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:247)
at
org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
at
org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
at
org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
at
org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
at
org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
at
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
at org.jboss.netty.channel.Channels.close(Channels.java:1065)
at
org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
at
org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
at
org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
at
org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
at
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
processing DATA.
java.io.IOException: Bad file descriptor
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:247)
at
org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
at
org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
at
org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
at
org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
...
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OOM with Netty
Posted by Norman Maurer <no...@apache.org>.
Thx Eric
Im lookin forward to your response
Bye
Norman
2010/4/25, Eric Charles <er...@u-mangate.com>:
> Hi Norman,
>
> Done.
> Wait and see.
>
> The good think this time is that we knew where to look at.
> Tks good logging.
>
> Tks for your very quick fixes.
>
> Eric
>
> PS : No file in /tmp but when I killed the process, 2 were left. So the
> /tmp files I reported came probably from the kill.
>
>
> On 04/24/2010 09:35 PM, Norman Maurer wrote:
>> Hi Eric,
>>
>> please svn up your source and try again.. I found at least one
>> possible problem which could cause a OOM.
>>
>> Thx,
>> Norman
>>
>>
>> 2010/4/24 Norman Maurer<no...@googlemail.com>:
>>
>>> Hi Eric,
>>>
>>> just downloading it..
>>>
>>> About the tread safety. I don't think so. I'm still wonder if the
>>> exception was only thrown because of the OOM or if its the cause of
>>> the OOM. Hopefully I will have a better view of the whole story once I
>>> looked at the heap dump...
>>>
>>> Bye,
>>> Norman
>>>
>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>
>>>> Hi Norman,
>>>>
>>>> You can download it from
>>>> http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip
>>>>
>>>>
>>>> "This error seems to get thrown when you write to a closed stream.."
>>>> Could it be a thread-safety problem ? (on thread using the session/state
>>>> of
>>>> another one ?)
>>>>
>>>> I will also monitor with lsof -p<pid> | grep deleted | wc -l
>>>>
>>>> As config change, I activated a few days ago the ValidRcptHandler
>>>> fastfail
>>>> handler.
>>>>
>>>> Tks,
>>>>
>>>> Eric
>>>>
>>>>
>>>> On 04/24/2010 03:13 PM, Norman Maurer wrote:
>>>>
>>>>> Could you upload the heap dump too ?
>>>>>
>>>>>
>>>>> 2010/4/24, Norman Maurer<no...@googlemail.com>:
>>>>>
>>>>>
>>>>>> Hi Eric,
>>>>>>
>>>>>> thx for the report. Comments are inline...
>>>>>>
>>>>>>
>>>>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>>>>
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I just got oom with netty impl.
>>>>>>> See http://apache.u-mangate.com/james/oom/20090424.png
>>>>>>> (http://apache.u-mangate.com/james/oom/index.html for history).
>>>>>>>
>>>>>>> Everything was fine with the trunk of last week that was really
>>>>>>> stable.
>>>>>>> I redeployed 3 days ago a new trunk version after a stop/start that
>>>>>>> showed
>>>>>>> ClassCastException of ImapMessage (ClassCastException was solved with
>>>>>>> the
>>>>>>> redeploy)
>>>>>>>
>>>>>>>
>>>>>> The ClassCastException should be fixed since yesterday. This was
>>>>>> caused by the EntityManager got confused by entities..
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Short infos about the today oom:
>>>>>>> - The logs had IOException "Bad file descriptor" (see copy of the
>>>>>>> exception
>>>>>>> after this mail)
>>>>>>>
>>>>>>>
>>>>>> This error seems to get thrown when you write to a closed stream...
>>>>>>
>>>>>>
>>>>>>
>>>>>>> - The begining time of these exceptions in the log is 00:10, and
>>>>>>> there
>>>>>>> are
>>>>>>> numerous ones during 1 minute.
>>>>>>> - The heap dump automatically generated via
>>>>>>> -XX:+HeapDumpOnOutOfMemoryError
>>>>>>> -XX:HeapDumpPath= occured on 00:10, exactly on the same time the
>>>>>>> IOException "Bad file descriptor" began.
>>>>>>>
>>>>>>>
>>>>>> Ok..
>>>>>>
>>>>>>
>>>>>>
>>>>>>> - I forgot to lsof to see the open files... but when if the error
>>>>>>> came
>>>>>>> from
>>>>>>> too many open files, the received exception in the log would have
>>>>>>> been
>>>>>>> "java.io.IOException: too many open files".
>>>>>>>
>>>>>>>
>>>>>> right..
>>>>>>
>>>>>>
>>>>>>
>>>>>>> - Seems like an infinite loop as Norman anticipated, but now
>>>>>>> stacktraces
>>>>>>> from logs seem to confirm this.
>>>>>>> - Looking at line 150 of
>>>>>>>
>>>>>>> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=rkup
>>>>>>> , out.write(line) seem to be the place that triggers the exception,
>>>>>>> out
>>>>>>> variable being =OutputStream)
>>>>>>> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
>>>>>>>
>>>>>>>
>>>>>> Good catch will see how this could happen..
>>>>>>
>>>>>>
>>>>>>
>>>>>>> - The server filesystem seems sane (new hardware, running with enough
>>>>>>> privilege, other applications running on the same machine without any
>>>>>>> problem). I don't see why I have a "Bad file descriptor"
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> See above..
>>>>>>
>>>>>>
>>>>>>
>>>>>>> There are file in tmp with creation date that span 1 week (so not
>>>>>>> produced
>>>>>>> on 00:10):
>>>>>>> - A few messagestream*.ms
>>>>>>> - More rewindable*.tmp
>>>>>>>
>>>>>>>
>>>>>> Will investigate ..
>>>>>>
>>>>>>
>>>>>>
>>>>>>> I restarted james.
>>>>>>> It still showed some issues (one account via pop3 gave errors in my
>>>>>>> client).
>>>>>>> oouch...
>>>>>>>
>>>>>>> I updated from trunk, redeployed, copied the var directory to migrate
>>>>>>> the
>>>>>>> account/domain/mail database, and restarted.
>>>>>>> Hopefully, now everything is working as before.
>>>>>>>
>>>>>>> So, we've got now a useful stacktrace that clearly shows where the
>>>>>>> oom
>>>>>>> "trigger" could be.
>>>>>>> I'm pretty sure that jconsole would have shown the same behaviour as
>>>>>>> http://apache.u-mangate.com/james/oom/oom.png
>>>>>>>
>>>>>>> Possible cause on the SMTP channel:
>>>>>>> - A weird behaviour of the smtp clients (hacking or respectfulness
>>>>>>> protocols
>>>>>>> clients)
>>>>>>> - A design in james causing infinite loop: thread clash (a thread
>>>>>>> accessing
>>>>>>> the same file as another thread?,...)
>>>>>>> - Additional James config (spamassassin or whatever) that leads to
>>>>>>> such
>>>>>>> behaviour
>>>>>>> - Hardware problem
>>>>>>>
>>>>>>> Currently, I run with default config + jdbcdomainlist.
>>>>>>> I except to receive another oom one of these days, unless the oom
>>>>>>> comes
>>>>>>> from
>>>>>>> an unactivated config (spamassassin or whatever).
>>>>>>>
>>>>>>> Tks,
>>>>>>>
>>>>>>> Eric
>>>>>>>
>>>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>>>>> processing DATA.
>>>>>>> java.io.IOException: Bad file descriptor
>>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
>>>>>>> at org.jboss.netty.channel.Channels.close(Channels.java:1065)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
>>>>>>> at
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>>>>> processing DATA.
>>>>>>> java.io.IOException: Bad file descriptor
>>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>> ...
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Bye,
>>>>>> Norman
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OOM with Netty
Posted by Eric Charles <er...@u-mangate.com>.
Hi Norman,
Done.
Wait and see.
The good think this time is that we knew where to look at.
Tks good logging.
Tks for your very quick fixes.
Eric
PS : No file in /tmp but when I killed the process, 2 were left. So the
/tmp files I reported came probably from the kill.
On 04/24/2010 09:35 PM, Norman Maurer wrote:
> Hi Eric,
>
> please svn up your source and try again.. I found at least one
> possible problem which could cause a OOM.
>
> Thx,
> Norman
>
>
> 2010/4/24 Norman Maurer<no...@googlemail.com>:
>
>> Hi Eric,
>>
>> just downloading it..
>>
>> About the tread safety. I don't think so. I'm still wonder if the
>> exception was only thrown because of the OOM or if its the cause of
>> the OOM. Hopefully I will have a better view of the whole story once I
>> looked at the heap dump...
>>
>> Bye,
>> Norman
>>
>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>
>>> Hi Norman,
>>>
>>> You can download it from
>>> http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip
>>>
>>>
>>> "This error seems to get thrown when you write to a closed stream.."
>>> Could it be a thread-safety problem ? (on thread using the session/state of
>>> another one ?)
>>>
>>> I will also monitor with lsof -p<pid> | grep deleted | wc -l
>>>
>>> As config change, I activated a few days ago the ValidRcptHandler fastfail
>>> handler.
>>>
>>> Tks,
>>>
>>> Eric
>>>
>>>
>>> On 04/24/2010 03:13 PM, Norman Maurer wrote:
>>>
>>>> Could you upload the heap dump too ?
>>>>
>>>>
>>>> 2010/4/24, Norman Maurer<no...@googlemail.com>:
>>>>
>>>>
>>>>> Hi Eric,
>>>>>
>>>>> thx for the report. Comments are inline...
>>>>>
>>>>>
>>>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>>>
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I just got oom with netty impl.
>>>>>> See http://apache.u-mangate.com/james/oom/20090424.png
>>>>>> (http://apache.u-mangate.com/james/oom/index.html for history).
>>>>>>
>>>>>> Everything was fine with the trunk of last week that was really stable.
>>>>>> I redeployed 3 days ago a new trunk version after a stop/start that
>>>>>> showed
>>>>>> ClassCastException of ImapMessage (ClassCastException was solved with
>>>>>> the
>>>>>> redeploy)
>>>>>>
>>>>>>
>>>>> The ClassCastException should be fixed since yesterday. This was
>>>>> caused by the EntityManager got confused by entities..
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Short infos about the today oom:
>>>>>> - The logs had IOException "Bad file descriptor" (see copy of the
>>>>>> exception
>>>>>> after this mail)
>>>>>>
>>>>>>
>>>>> This error seems to get thrown when you write to a closed stream...
>>>>>
>>>>>
>>>>>
>>>>>> - The begining time of these exceptions in the log is 00:10, and there
>>>>>> are
>>>>>> numerous ones during 1 minute.
>>>>>> - The heap dump automatically generated via
>>>>>> -XX:+HeapDumpOnOutOfMemoryError
>>>>>> -XX:HeapDumpPath= occured on 00:10, exactly on the same time the
>>>>>> IOException "Bad file descriptor" began.
>>>>>>
>>>>>>
>>>>> Ok..
>>>>>
>>>>>
>>>>>
>>>>>> - I forgot to lsof to see the open files... but when if the error came
>>>>>> from
>>>>>> too many open files, the received exception in the log would have been
>>>>>> "java.io.IOException: too many open files".
>>>>>>
>>>>>>
>>>>> right..
>>>>>
>>>>>
>>>>>
>>>>>> - Seems like an infinite loop as Norman anticipated, but now stacktraces
>>>>>> from logs seem to confirm this.
>>>>>> - Looking at line 150 of
>>>>>>
>>>>>> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=rkup
>>>>>> , out.write(line) seem to be the place that triggers the exception, out
>>>>>> variable being =OutputStream)
>>>>>> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
>>>>>>
>>>>>>
>>>>> Good catch will see how this could happen..
>>>>>
>>>>>
>>>>>
>>>>>> - The server filesystem seems sane (new hardware, running with enough
>>>>>> privilege, other applications running on the same machine without any
>>>>>> problem). I don't see why I have a "Bad file descriptor"
>>>>>>
>>>>>>
>>>>>>
>>>>> See above..
>>>>>
>>>>>
>>>>>
>>>>>> There are file in tmp with creation date that span 1 week (so not
>>>>>> produced
>>>>>> on 00:10):
>>>>>> - A few messagestream*.ms
>>>>>> - More rewindable*.tmp
>>>>>>
>>>>>>
>>>>> Will investigate ..
>>>>>
>>>>>
>>>>>
>>>>>> I restarted james.
>>>>>> It still showed some issues (one account via pop3 gave errors in my
>>>>>> client).
>>>>>> oouch...
>>>>>>
>>>>>> I updated from trunk, redeployed, copied the var directory to migrate
>>>>>> the
>>>>>> account/domain/mail database, and restarted.
>>>>>> Hopefully, now everything is working as before.
>>>>>>
>>>>>> So, we've got now a useful stacktrace that clearly shows where the oom
>>>>>> "trigger" could be.
>>>>>> I'm pretty sure that jconsole would have shown the same behaviour as
>>>>>> http://apache.u-mangate.com/james/oom/oom.png
>>>>>>
>>>>>> Possible cause on the SMTP channel:
>>>>>> - A weird behaviour of the smtp clients (hacking or respectfulness
>>>>>> protocols
>>>>>> clients)
>>>>>> - A design in james causing infinite loop: thread clash (a thread
>>>>>> accessing
>>>>>> the same file as another thread?,...)
>>>>>> - Additional James config (spamassassin or whatever) that leads to such
>>>>>> behaviour
>>>>>> - Hardware problem
>>>>>>
>>>>>> Currently, I run with default config + jdbcdomainlist.
>>>>>> I except to receive another oom one of these days, unless the oom comes
>>>>>> from
>>>>>> an unactivated config (spamassassin or whatever).
>>>>>>
>>>>>> Tks,
>>>>>>
>>>>>> Eric
>>>>>>
>>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>>>> processing DATA.
>>>>>> java.io.IOException: Bad file descriptor
>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
>>>>>> at org.jboss.netty.channel.Channels.close(Channels.java:1065)
>>>>>> at
>>>>>> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>> at
>>>>>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>> at
>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
>>>>>> at
>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
>>>>>> at
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>>>> processing DATA.
>>>>>> java.io.IOException: Bad file descriptor
>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>> ...
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> Bye,
>>>>> Norman
>>>>>
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OOM with Netty
Posted by Norman Maurer <no...@googlemail.com>.
Hi Eric,
please svn up your source and try again.. I found at least one
possible problem which could cause a OOM.
Thx,
Norman
2010/4/24 Norman Maurer <no...@googlemail.com>:
> Hi Eric,
>
> just downloading it..
>
> About the tread safety. I don't think so. I'm still wonder if the
> exception was only thrown because of the OOM or if its the cause of
> the OOM. Hopefully I will have a better view of the whole story once I
> looked at the heap dump...
>
> Bye,
> Norman
>
> 2010/4/24 Eric Charles <er...@u-mangate.com>:
>> Hi Norman,
>>
>> You can download it from
>> http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip
>>
>>
>> "This error seems to get thrown when you write to a closed stream.."
>> Could it be a thread-safety problem ? (on thread using the session/state of
>> another one ?)
>>
>> I will also monitor with lsof -p <pid> | grep deleted | wc -l
>>
>> As config change, I activated a few days ago the ValidRcptHandler fastfail
>> handler.
>>
>> Tks,
>>
>> Eric
>>
>>
>> On 04/24/2010 03:13 PM, Norman Maurer wrote:
>>>
>>> Could you upload the heap dump too ?
>>>
>>>
>>> 2010/4/24, Norman Maurer<no...@googlemail.com>:
>>>
>>>>
>>>> Hi Eric,
>>>>
>>>> thx for the report. Comments are inline...
>>>>
>>>>
>>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>>
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I just got oom with netty impl.
>>>>> See http://apache.u-mangate.com/james/oom/20090424.png
>>>>> (http://apache.u-mangate.com/james/oom/index.html for history).
>>>>>
>>>>> Everything was fine with the trunk of last week that was really stable.
>>>>> I redeployed 3 days ago a new trunk version after a stop/start that
>>>>> showed
>>>>> ClassCastException of ImapMessage (ClassCastException was solved with
>>>>> the
>>>>> redeploy)
>>>>>
>>>>
>>>> The ClassCastException should be fixed since yesterday. This was
>>>> caused by the EntityManager got confused by entities..
>>>>
>>>>
>>>>
>>>>>
>>>>> Short infos about the today oom:
>>>>> - The logs had IOException "Bad file descriptor" (see copy of the
>>>>> exception
>>>>> after this mail)
>>>>>
>>>>
>>>> This error seems to get thrown when you write to a closed stream...
>>>>
>>>>
>>>>>
>>>>> - The begining time of these exceptions in the log is 00:10, and there
>>>>> are
>>>>> numerous ones during 1 minute.
>>>>> - The heap dump automatically generated via
>>>>> -XX:+HeapDumpOnOutOfMemoryError
>>>>> -XX:HeapDumpPath=/ occured on 00:10, exactly on the same time the
>>>>> IOException "Bad file descriptor" began.
>>>>>
>>>>
>>>> Ok..
>>>>
>>>>
>>>>>
>>>>> - I forgot to lsof to see the open files... but when if the error came
>>>>> from
>>>>> too many open files, the received exception in the log would have been
>>>>> "java.io.IOException: too many open files".
>>>>>
>>>>
>>>> right..
>>>>
>>>>
>>>>>
>>>>> - Seems like an infinite loop as Norman anticipated, but now stacktraces
>>>>> from logs seem to confirm this.
>>>>> - Looking at line 150 of
>>>>>
>>>>> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=markup
>>>>> , out.write(line) seem to be the place that triggers the exception, out
>>>>> variable being = (OutputStream)
>>>>> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
>>>>>
>>>>
>>>> Good catch will see how this could happen..
>>>>
>>>>
>>>>>
>>>>> - The server filesystem seems sane (new hardware, running with enough
>>>>> privilege, other applications running on the same machine without any
>>>>> problem). I don't see why I have a "Bad file descriptor"
>>>>>
>>>>>
>>>>
>>>> See above..
>>>>
>>>>
>>>>>
>>>>> There are file in tmp with creation date that span 1 week (so not
>>>>> produced
>>>>> on 00:10):
>>>>> - A few messagestream*.ms
>>>>> - More rewindable*.tmp
>>>>>
>>>>
>>>> Will investigate ..
>>>>
>>>>
>>>>>
>>>>> I restarted james.
>>>>> It still showed some issues (one account via pop3 gave errors in my
>>>>> client).
>>>>> oouch...
>>>>>
>>>>> I updated from trunk, redeployed, copied the var directory to migrate
>>>>> the
>>>>> account/domain/mail database, and restarted.
>>>>> Hopefully, now everything is working as before.
>>>>>
>>>>> So, we've got now a useful stacktrace that clearly shows where the oom
>>>>> "trigger" could be.
>>>>> I'm pretty sure that jconsole would have shown the same behaviour as
>>>>> http://apache.u-mangate.com/james/oom/oom.png
>>>>>
>>>>> Possible cause on the SMTP channel:
>>>>> - A weird behaviour of the smtp clients (hacking or respectfulness
>>>>> protocols
>>>>> clients)
>>>>> - A design in james causing infinite loop: thread clash (a thread
>>>>> accessing
>>>>> the same file as another thread?,...)
>>>>> - Additional James config (spamassassin or whatever) that leads to such
>>>>> behaviour
>>>>> - Hardware problem
>>>>>
>>>>> Currently, I run with default config + jdbcdomainlist.
>>>>> I except to receive another oom one of these days, unless the oom comes
>>>>> from
>>>>> an unactivated config (spamassassin or whatever).
>>>>>
>>>>> Tks,
>>>>>
>>>>> Eric
>>>>>
>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>>> processing DATA.
>>>>> java.io.IOException: Bad file descriptor
>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>> at
>>>>>
>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>> at
>>>>>
>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
>>>>> at org.jboss.netty.channel.Channels.close(Channels.java:1065)
>>>>> at
>>>>> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
>>>>> at
>>>>>
>>>>> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>> at
>>>>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>> at
>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
>>>>> at
>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>>> processing DATA.
>>>>> java.io.IOException: Bad file descriptor
>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>> at
>>>>>
>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>> at
>>>>>
>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>> ...
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>>> Bye,
>>>> Norman
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Clean shutdown
Posted by Eric Charles <er...@u-mangate.com>.
Hi Norman,
I will open a jira.
Do you see the implementation as a a parametrizable main method in the
http://svn.apache.org/viewvc/james/server/trunk/spring-deployment/src/main/java/org/apache/james/container/spring/Main.java?view=markup
?
The scripts would pass start or stop parameter and depending on this
parameter, the main will do its job.
Right?
Eric
On 04/24/2010 10:13 PM, Norman Maurer wrote:
> Hi Eric,
>
> I think the used name of "shutdown" is correct. About the start /
> stop script you are right. It would be very nice to have this. Could
> you open a jira issue for it ? I think the maven appassember plugin
> (http://mojo.codehaus.org/appassembler/appassembler-maven-plugin/)
> could help with it and do all the magic for us..
>
> Thx ,
> Norman
>
> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>
>> mmh, as requirement to start/stop james, I would like to read java 1.? in
>> the doc :)
>>
>> Would you agree I open a jira for this (rename shutdown to kill and create a
>> clean shutdown).
>> I could also try to contribute a patch for this.
>>
>> Tks,
>>
>> Eric
>>
>>
>> On 04/24/2010 06:34 PM, Norman Maurer wrote:
>>
>>> Hi Eric,
>>>
>>> you should use "screen" (http://www.gnu.org/software/screen/) for
>>> launching james ;)
>>>
>>> RemotManager would be the same as kill<pid>
>>>
>>> Bye,
>>> Norman
>>>
>>>
>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>
>>>
>>>> Hi Norman,
>>>> I launch with& (background process) so I can leave the terminal.
>>>> Would shutdown via remotemanager better ?
>>>> Tks,
>>>> Eric
>>>>
>>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Clean shutdown (was: OOM with Netty)
Posted by Norman Maurer <no...@googlemail.com>.
Hi Eric,
I think the used name of "shutdown" is correct. About the start /
stop script you are right. It would be very nice to have this. Could
you open a jira issue for it ? I think the maven appassember plugin
(http://mojo.codehaus.org/appassembler/appassembler-maven-plugin/)
could help with it and do all the magic for us..
Thx ,
Norman
2010/4/24 Eric Charles <er...@u-mangate.com>:
> mmh, as requirement to start/stop james, I would like to read java 1.? in
> the doc :)
>
> Would you agree I open a jira for this (rename shutdown to kill and create a
> clean shutdown).
> I could also try to contribute a patch for this.
>
> Tks,
>
> Eric
>
>
> On 04/24/2010 06:34 PM, Norman Maurer wrote:
>>
>> Hi Eric,
>>
>> you should use "screen" (http://www.gnu.org/software/screen/) for
>> launching james ;)
>>
>> RemotManager would be the same as kill<pid>
>>
>> Bye,
>> Norman
>>
>>
>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>
>>>
>>> Hi Norman,
>>> I launch with& (background process) so I can leave the terminal.
>>> Would shutdown via remotemanager better ?
>>> Tks,
>>> Eric
>>>
>>>
>>> On 04/24/2010 06:19 PM, Norman Maurer wrote:
>>>
>>>>
>>>> Hi Eric,
>>>>
>>>> doesn't CTRL-C work in the terminal you started james in ? And you
>>>> should not use kill-9<pid> because it will not let JAMES cleanup
>>>> stuff, if you want to use kill<pid> ..
>>>>
>>>> Bye,
>>>> Norman
>>>>
>>>>
>>>>
>>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>>
>>>>
>>>>>
>>>>> Hi Norman,
>>>>>
>>>>> lsof -p<pid> | grep deleted | wc -l gives a stable number (around
>>>>> 20,
>>>>> growing and after decreasing : cleanup of gc or os... whatever...).
>>>>>
>>>>> No file in /tmp.
>>>>>
>>>>> I wonder if the files I found were files left further to the few
>>>>> "hard-stop"
>>>>> I made during last week.
>>>>> Currently, to stop James, I use kill -9<pid>
>>>>>
>>>>> Is there another cleaner way to do this ?
>>>>> I saw shutdown command ("kills the current JVM (convenient when James
>>>>> is
>>>>> run as a daemon)") in the remotemanager.
>>>>> Is it different from kill -9 ? Should I use this ?
>>>>>
>>>>> Tks,
>>>>>
>>>>> Eric
>>>>>
>>>>>
>>>>>
>>>>> On 04/24/2010 04:23 PM, Norman Maurer wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> Hi Eric,
>>>>>>
>>>>>> just downloading it..
>>>>>>
>>>>>> About the tread safety. I don't think so. I'm still wonder if the
>>>>>> exception was only thrown because of the OOM or if its the cause of
>>>>>> the OOM. Hopefully I will have a better view of the whole story once I
>>>>>> looked at the heap dump...
>>>>>>
>>>>>> Bye,
>>>>>> Norman
>>>>>>
>>>>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Hi Norman,
>>>>>>>
>>>>>>> You can download it from
>>>>>>> http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip
>>>>>>>
>>>>>>>
>>>>>>> "This error seems to get thrown when you write to a closed stream.."
>>>>>>> Could it be a thread-safety problem ? (on thread using the
>>>>>>> session/state
>>>>>>> of
>>>>>>> another one ?)
>>>>>>>
>>>>>>> I will also monitor with lsof -p<pid> | grep deleted | wc -l
>>>>>>>
>>>>>>> As config change, I activated a few days ago the ValidRcptHandler
>>>>>>> fastfail
>>>>>>> handler.
>>>>>>>
>>>>>>> Tks,
>>>>>>>
>>>>>>> Eric
>>>>>>>
>>>>>>>
>>>>>>> On 04/24/2010 03:13 PM, Norman Maurer wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Could you upload the heap dump too ?
>>>>>>>>
>>>>>>>>
>>>>>>>> 2010/4/24, Norman Maurer<no...@googlemail.com>:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Eric,
>>>>>>>>>
>>>>>>>>> thx for the report. Comments are inline...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> I just got oom with netty impl.
>>>>>>>>>> See http://apache.u-mangate.com/james/oom/20090424.png
>>>>>>>>>> (http://apache.u-mangate.com/james/oom/index.html for history).
>>>>>>>>>>
>>>>>>>>>> Everything was fine with the trunk of last week that was really
>>>>>>>>>> stable.
>>>>>>>>>> I redeployed 3 days ago a new trunk version after a stop/start
>>>>>>>>>> that
>>>>>>>>>> showed
>>>>>>>>>> ClassCastException of ImapMessage (ClassCastException was solved
>>>>>>>>>> with
>>>>>>>>>> the
>>>>>>>>>> redeploy)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The ClassCastException should be fixed since yesterday. This was
>>>>>>>>> caused by the EntityManager got confused by entities..
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Short infos about the today oom:
>>>>>>>>>> - The logs had IOException "Bad file descriptor" (see copy of the
>>>>>>>>>> exception
>>>>>>>>>> after this mail)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This error seems to get thrown when you write to a closed stream...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - The begining time of these exceptions in the log is 00:10, and
>>>>>>>>>> there
>>>>>>>>>> are
>>>>>>>>>> numerous ones during 1 minute.
>>>>>>>>>> - The heap dump automatically generated via
>>>>>>>>>> -XX:+HeapDumpOnOutOfMemoryError
>>>>>>>>>> -XX:HeapDumpPathÌured on 00:10, exactly on the same time the
>>>>>>>>>> IOException "Bad file descriptor" began.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ok..
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - I forgot to lsof to see the open files... but when if the error
>>>>>>>>>> came
>>>>>>>>>> from
>>>>>>>>>> too many open files, the received exception in the log would have
>>>>>>>>>> been
>>>>>>>>>> "java.io.IOException: too many open files".
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> right..
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - Seems like an infinite loop as Norman anticipated, but now
>>>>>>>>>> stacktraces
>>>>>>>>>> from logs seem to confirm this.
>>>>>>>>>> - Looking at line 150 of
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=
>>>>>>>>>> , out.write(line) seem to be the place that triggers the
>>>>>>>>>> exception,
>>>>>>>>>> out
>>>>>>>>>> variable being =utStream)
>>>>>>>>>>
>>>>>>>>>> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Good catch will see how this could happen..
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - The server filesystem seems sane (new hardware, running with
>>>>>>>>>> enough
>>>>>>>>>> privilege, other applications running on the same machine without
>>>>>>>>>> any
>>>>>>>>>> problem). I don't see why I have a "Bad file descriptor"
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> See above..
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> There are file in tmp with creation date that span 1 week (so not
>>>>>>>>>> produced
>>>>>>>>>> on 00:10):
>>>>>>>>>> - A few messagestream*.ms
>>>>>>>>>> - More rewindable*.tmp
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Will investigate ..
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I restarted james.
>>>>>>>>>> It still showed some issues (one account via pop3 gave errors in
>>>>>>>>>> my
>>>>>>>>>> client).
>>>>>>>>>> oouch...
>>>>>>>>>>
>>>>>>>>>> I updated from trunk, redeployed, copied the var directory to
>>>>>>>>>> migrate
>>>>>>>>>> the
>>>>>>>>>> account/domain/mail database, and restarted.
>>>>>>>>>> Hopefully, now everything is working as before.
>>>>>>>>>>
>>>>>>>>>> So, we've got now a useful stacktrace that clearly shows where the
>>>>>>>>>> oom
>>>>>>>>>> "trigger" could be.
>>>>>>>>>> I'm pretty sure that jconsole would have shown the same behaviour
>>>>>>>>>> as
>>>>>>>>>> http://apache.u-mangate.com/james/oom/oom.png
>>>>>>>>>>
>>>>>>>>>> Possible cause on the SMTP channel:
>>>>>>>>>> - A weird behaviour of the smtp clients (hacking or respectfulness
>>>>>>>>>> protocols
>>>>>>>>>> clients)
>>>>>>>>>> - A design in james causing infinite loop: thread clash (a thread
>>>>>>>>>> accessing
>>>>>>>>>> the same file as another thread?,...)
>>>>>>>>>> - Additional James config (spamassassin or whatever) that leads to
>>>>>>>>>> such
>>>>>>>>>> behaviour
>>>>>>>>>> - Hardware problem
>>>>>>>>>>
>>>>>>>>>> Currently, I run with default config + jdbcdomainlist.
>>>>>>>>>> I except to receive another oom one of these days, unless the oom
>>>>>>>>>> comes
>>>>>>>>>> from
>>>>>>>>>> an unactivated config (spamassassin or whatever).
>>>>>>>>>>
>>>>>>>>>> Tks,
>>>>>>>>>>
>>>>>>>>>> Eric
>>>>>>>>>>
>>>>>>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred
>>>>>>>>>> while
>>>>>>>>>> processing DATA.
>>>>>>>>>> java.io.IOException: Bad file descriptor
>>>>>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
>>>>>>>>>> at org.jboss.netty.channel.Channels.close(Channels.java:1065)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred
>>>>>>>>>> while
>>>>>>>>>> processing DATA.
>>>>>>>>>> java.io.IOException: Bad file descriptor
>>>>>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Bye,
>>>>>>>>> Norman
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Clean shutdown (was: OOM with Netty)
Posted by Eric Charles <er...@u-mangate.com>.
mmh, as requirement to start/stop james, I would like to read java 1.?
in the doc :)
Would you agree I open a jira for this (rename shutdown to kill and
create a clean shutdown).
I could also try to contribute a patch for this.
Tks,
Eric
On 04/24/2010 06:34 PM, Norman Maurer wrote:
> Hi Eric,
>
> you should use "screen" (http://www.gnu.org/software/screen/) for
> launching james ;)
>
> RemotManager would be the same as kill<pid>
>
> Bye,
> Norman
>
>
> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>
>> Hi Norman,
>> I launch with& (background process) so I can leave the terminal.
>> Would shutdown via remotemanager better ?
>> Tks,
>> Eric
>>
>>
>> On 04/24/2010 06:19 PM, Norman Maurer wrote:
>>
>>> Hi Eric,
>>>
>>> doesn't CTRL-C work in the terminal you started james in ? And you
>>> should not use kill-9<pid> because it will not let JAMES cleanup
>>> stuff, if you want to use kill<pid> ..
>>>
>>> Bye,
>>> Norman
>>>
>>>
>>>
>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>
>>>
>>>> Hi Norman,
>>>>
>>>> lsof -p<pid> | grep deleted | wc -l gives a stable number (around 20,
>>>> growing and after decreasing : cleanup of gc or os... whatever...).
>>>>
>>>> No file in /tmp.
>>>>
>>>> I wonder if the files I found were files left further to the few
>>>> "hard-stop"
>>>> I made during last week.
>>>> Currently, to stop James, I use kill -9<pid>
>>>>
>>>> Is there another cleaner way to do this ?
>>>> I saw shutdown command ("kills the current JVM (convenient when James is
>>>> run as a daemon)") in the remotemanager.
>>>> Is it different from kill -9 ? Should I use this ?
>>>>
>>>> Tks,
>>>>
>>>> Eric
>>>>
>>>>
>>>>
>>>> On 04/24/2010 04:23 PM, Norman Maurer wrote:
>>>>
>>>>
>>>>> Hi Eric,
>>>>>
>>>>> just downloading it..
>>>>>
>>>>> About the tread safety. I don't think so. I'm still wonder if the
>>>>> exception was only thrown because of the OOM or if its the cause of
>>>>> the OOM. Hopefully I will have a better view of the whole story once I
>>>>> looked at the heap dump...
>>>>>
>>>>> Bye,
>>>>> Norman
>>>>>
>>>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>>>
>>>>>
>>>>>
>>>>>> Hi Norman,
>>>>>>
>>>>>> You can download it from
>>>>>> http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip
>>>>>>
>>>>>>
>>>>>> "This error seems to get thrown when you write to a closed stream.."
>>>>>> Could it be a thread-safety problem ? (on thread using the
>>>>>> session/state
>>>>>> of
>>>>>> another one ?)
>>>>>>
>>>>>> I will also monitor with lsof -p<pid> | grep deleted | wc -l
>>>>>>
>>>>>> As config change, I activated a few days ago the ValidRcptHandler
>>>>>> fastfail
>>>>>> handler.
>>>>>>
>>>>>> Tks,
>>>>>>
>>>>>> Eric
>>>>>>
>>>>>>
>>>>>> On 04/24/2010 03:13 PM, Norman Maurer wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Could you upload the heap dump too ?
>>>>>>>
>>>>>>>
>>>>>>> 2010/4/24, Norman Maurer<no...@googlemail.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Hi Eric,
>>>>>>>>
>>>>>>>> thx for the report. Comments are inline...
>>>>>>>>
>>>>>>>>
>>>>>>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> I just got oom with netty impl.
>>>>>>>>> See http://apache.u-mangate.com/james/oom/20090424.png
>>>>>>>>> (http://apache.u-mangate.com/james/oom/index.html for history).
>>>>>>>>>
>>>>>>>>> Everything was fine with the trunk of last week that was really
>>>>>>>>> stable.
>>>>>>>>> I redeployed 3 days ago a new trunk version after a stop/start that
>>>>>>>>> showed
>>>>>>>>> ClassCastException of ImapMessage (ClassCastException was solved
>>>>>>>>> with
>>>>>>>>> the
>>>>>>>>> redeploy)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> The ClassCastException should be fixed since yesterday. This was
>>>>>>>> caused by the EntityManager got confused by entities..
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Short infos about the today oom:
>>>>>>>>> - The logs had IOException "Bad file descriptor" (see copy of the
>>>>>>>>> exception
>>>>>>>>> after this mail)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> This error seems to get thrown when you write to a closed stream...
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> - The begining time of these exceptions in the log is 00:10, and
>>>>>>>>> there
>>>>>>>>> are
>>>>>>>>> numerous ones during 1 minute.
>>>>>>>>> - The heap dump automatically generated via
>>>>>>>>> -XX:+HeapDumpOnOutOfMemoryError
>>>>>>>>> -XX:HeapDumpPathÌured on 00:10, exactly on the same time the
>>>>>>>>> IOException "Bad file descriptor" began.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Ok..
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> - I forgot to lsof to see the open files... but when if the error
>>>>>>>>> came
>>>>>>>>> from
>>>>>>>>> too many open files, the received exception in the log would have
>>>>>>>>> been
>>>>>>>>> "java.io.IOException: too many open files".
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> right..
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> - Seems like an infinite loop as Norman anticipated, but now
>>>>>>>>> stacktraces
>>>>>>>>> from logs seem to confirm this.
>>>>>>>>> - Looking at line 150 of
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=
>>>>>>>>> , out.write(line) seem to be the place that triggers the exception,
>>>>>>>>> out
>>>>>>>>> variable being =utStream)
>>>>>>>>> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Good catch will see how this could happen..
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> - The server filesystem seems sane (new hardware, running with
>>>>>>>>> enough
>>>>>>>>> privilege, other applications running on the same machine without
>>>>>>>>> any
>>>>>>>>> problem). I don't see why I have a "Bad file descriptor"
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> See above..
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> There are file in tmp with creation date that span 1 week (so not
>>>>>>>>> produced
>>>>>>>>> on 00:10):
>>>>>>>>> - A few messagestream*.ms
>>>>>>>>> - More rewindable*.tmp
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Will investigate ..
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> I restarted james.
>>>>>>>>> It still showed some issues (one account via pop3 gave errors in my
>>>>>>>>> client).
>>>>>>>>> oouch...
>>>>>>>>>
>>>>>>>>> I updated from trunk, redeployed, copied the var directory to
>>>>>>>>> migrate
>>>>>>>>> the
>>>>>>>>> account/domain/mail database, and restarted.
>>>>>>>>> Hopefully, now everything is working as before.
>>>>>>>>>
>>>>>>>>> So, we've got now a useful stacktrace that clearly shows where the
>>>>>>>>> oom
>>>>>>>>> "trigger" could be.
>>>>>>>>> I'm pretty sure that jconsole would have shown the same behaviour as
>>>>>>>>> http://apache.u-mangate.com/james/oom/oom.png
>>>>>>>>>
>>>>>>>>> Possible cause on the SMTP channel:
>>>>>>>>> - A weird behaviour of the smtp clients (hacking or respectfulness
>>>>>>>>> protocols
>>>>>>>>> clients)
>>>>>>>>> - A design in james causing infinite loop: thread clash (a thread
>>>>>>>>> accessing
>>>>>>>>> the same file as another thread?,...)
>>>>>>>>> - Additional James config (spamassassin or whatever) that leads to
>>>>>>>>> such
>>>>>>>>> behaviour
>>>>>>>>> - Hardware problem
>>>>>>>>>
>>>>>>>>> Currently, I run with default config + jdbcdomainlist.
>>>>>>>>> I except to receive another oom one of these days, unless the oom
>>>>>>>>> comes
>>>>>>>>> from
>>>>>>>>> an unactivated config (spamassassin or whatever).
>>>>>>>>>
>>>>>>>>> Tks,
>>>>>>>>>
>>>>>>>>> Eric
>>>>>>>>>
>>>>>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>>>>>>> processing DATA.
>>>>>>>>> java.io.IOException: Bad file descriptor
>>>>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
>>>>>>>>> at org.jboss.netty.channel.Channels.close(Channels.java:1065)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>>>>>>> processing DATA.
>>>>>>>>> java.io.IOException: Bad file descriptor
>>>>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Bye,
>>>>>>>> Norman
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OOM with Netty
Posted by Norman Maurer <no...@googlemail.com>.
Hi Eric,
you should use "screen" (http://www.gnu.org/software/screen/) for
launching james ;)
RemotManager would be the same as kill <pid>
Bye,
Norman
2010/4/24 Eric Charles <er...@u-mangate.com>:
> Hi Norman,
> I launch with & (background process) so I can leave the terminal.
> Would shutdown via remotemanager better ?
> Tks,
> Eric
>
>
> On 04/24/2010 06:19 PM, Norman Maurer wrote:
>>
>> Hi Eric,
>>
>> doesn't CTRL-C work in the terminal you started james in ? And you
>> should not use kill-9<pid> because it will not let JAMES cleanup
>> stuff, if you want to use kill<pid> ..
>>
>> Bye,
>> Norman
>>
>>
>>
>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>
>>>
>>> Hi Norman,
>>>
>>> lsof -p<pid> | grep deleted | wc -l gives a stable number (around 20,
>>> growing and after decreasing : cleanup of gc or os... whatever...).
>>>
>>> No file in /tmp.
>>>
>>> I wonder if the files I found were files left further to the few
>>> "hard-stop"
>>> I made during last week.
>>> Currently, to stop James, I use kill -9<pid>
>>>
>>> Is there another cleaner way to do this ?
>>> I saw shutdown command ("kills the current JVM (convenient when James is
>>> run as a daemon)") in the remotemanager.
>>> Is it different from kill -9 ? Should I use this ?
>>>
>>> Tks,
>>>
>>> Eric
>>>
>>>
>>>
>>> On 04/24/2010 04:23 PM, Norman Maurer wrote:
>>>
>>>>
>>>> Hi Eric,
>>>>
>>>> just downloading it..
>>>>
>>>> About the tread safety. I don't think so. I'm still wonder if the
>>>> exception was only thrown because of the OOM or if its the cause of
>>>> the OOM. Hopefully I will have a better view of the whole story once I
>>>> looked at the heap dump...
>>>>
>>>> Bye,
>>>> Norman
>>>>
>>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>>
>>>>
>>>>>
>>>>> Hi Norman,
>>>>>
>>>>> You can download it from
>>>>> http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip
>>>>>
>>>>>
>>>>> "This error seems to get thrown when you write to a closed stream.."
>>>>> Could it be a thread-safety problem ? (on thread using the
>>>>> session/state
>>>>> of
>>>>> another one ?)
>>>>>
>>>>> I will also monitor with lsof -p<pid> | grep deleted | wc -l
>>>>>
>>>>> As config change, I activated a few days ago the ValidRcptHandler
>>>>> fastfail
>>>>> handler.
>>>>>
>>>>> Tks,
>>>>>
>>>>> Eric
>>>>>
>>>>>
>>>>> On 04/24/2010 03:13 PM, Norman Maurer wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> Could you upload the heap dump too ?
>>>>>>
>>>>>>
>>>>>> 2010/4/24, Norman Maurer<no...@googlemail.com>:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Hi Eric,
>>>>>>>
>>>>>>> thx for the report. Comments are inline...
>>>>>>>
>>>>>>>
>>>>>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> I just got oom with netty impl.
>>>>>>>> See http://apache.u-mangate.com/james/oom/20090424.png
>>>>>>>> (http://apache.u-mangate.com/james/oom/index.html for history).
>>>>>>>>
>>>>>>>> Everything was fine with the trunk of last week that was really
>>>>>>>> stable.
>>>>>>>> I redeployed 3 days ago a new trunk version after a stop/start that
>>>>>>>> showed
>>>>>>>> ClassCastException of ImapMessage (ClassCastException was solved
>>>>>>>> with
>>>>>>>> the
>>>>>>>> redeploy)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> The ClassCastException should be fixed since yesterday. This was
>>>>>>> caused by the EntityManager got confused by entities..
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Short infos about the today oom:
>>>>>>>> - The logs had IOException "Bad file descriptor" (see copy of the
>>>>>>>> exception
>>>>>>>> after this mail)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> This error seems to get thrown when you write to a closed stream...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> - The begining time of these exceptions in the log is 00:10, and
>>>>>>>> there
>>>>>>>> are
>>>>>>>> numerous ones during 1 minute.
>>>>>>>> - The heap dump automatically generated via
>>>>>>>> -XX:+HeapDumpOnOutOfMemoryError
>>>>>>>> -XX:HeapDumpPath=ccured on 00:10, exactly on the same time the
>>>>>>>> IOException "Bad file descriptor" began.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> Ok..
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> - I forgot to lsof to see the open files... but when if the error
>>>>>>>> came
>>>>>>>> from
>>>>>>>> too many open files, the received exception in the log would have
>>>>>>>> been
>>>>>>>> "java.io.IOException: too many open files".
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> right..
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> - Seems like an infinite loop as Norman anticipated, but now
>>>>>>>> stacktraces
>>>>>>>> from logs seem to confirm this.
>>>>>>>> - Looking at line 150 of
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=up
>>>>>>>> , out.write(line) seem to be the place that triggers the exception,
>>>>>>>> out
>>>>>>>> variable being =tputStream)
>>>>>>>> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> Good catch will see how this could happen..
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> - The server filesystem seems sane (new hardware, running with
>>>>>>>> enough
>>>>>>>> privilege, other applications running on the same machine without
>>>>>>>> any
>>>>>>>> problem). I don't see why I have a "Bad file descriptor"
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> See above..
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> There are file in tmp with creation date that span 1 week (so not
>>>>>>>> produced
>>>>>>>> on 00:10):
>>>>>>>> - A few messagestream*.ms
>>>>>>>> - More rewindable*.tmp
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> Will investigate ..
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> I restarted james.
>>>>>>>> It still showed some issues (one account via pop3 gave errors in my
>>>>>>>> client).
>>>>>>>> oouch...
>>>>>>>>
>>>>>>>> I updated from trunk, redeployed, copied the var directory to
>>>>>>>> migrate
>>>>>>>> the
>>>>>>>> account/domain/mail database, and restarted.
>>>>>>>> Hopefully, now everything is working as before.
>>>>>>>>
>>>>>>>> So, we've got now a useful stacktrace that clearly shows where the
>>>>>>>> oom
>>>>>>>> "trigger" could be.
>>>>>>>> I'm pretty sure that jconsole would have shown the same behaviour as
>>>>>>>> http://apache.u-mangate.com/james/oom/oom.png
>>>>>>>>
>>>>>>>> Possible cause on the SMTP channel:
>>>>>>>> - A weird behaviour of the smtp clients (hacking or respectfulness
>>>>>>>> protocols
>>>>>>>> clients)
>>>>>>>> - A design in james causing infinite loop: thread clash (a thread
>>>>>>>> accessing
>>>>>>>> the same file as another thread?,...)
>>>>>>>> - Additional James config (spamassassin or whatever) that leads to
>>>>>>>> such
>>>>>>>> behaviour
>>>>>>>> - Hardware problem
>>>>>>>>
>>>>>>>> Currently, I run with default config + jdbcdomainlist.
>>>>>>>> I except to receive another oom one of these days, unless the oom
>>>>>>>> comes
>>>>>>>> from
>>>>>>>> an unactivated config (spamassassin or whatever).
>>>>>>>>
>>>>>>>> Tks,
>>>>>>>>
>>>>>>>> Eric
>>>>>>>>
>>>>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>>>>>> processing DATA.
>>>>>>>> java.io.IOException: Bad file descriptor
>>>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
>>>>>>>> at org.jboss.netty.channel.Channels.close(Channels.java:1065)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>>>>>> processing DATA.
>>>>>>>> java.io.IOException: Bad file descriptor
>>>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>>> ...
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> Bye,
>>>>>>> Norman
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OOM with Netty
Posted by Eric Charles <er...@u-mangate.com>.
Hi Norman,
I launch with & (background process) so I can leave the terminal.
Would shutdown via remotemanager better ?
Tks,
Eric
On 04/24/2010 06:19 PM, Norman Maurer wrote:
> Hi Eric,
>
> doesn't CTRL-C work in the terminal you started james in ? And you
> should not use kill-9<pid> because it will not let JAMES cleanup
> stuff, if you want to use kill<pid> ..
>
> Bye,
> Norman
>
>
>
> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>
>> Hi Norman,
>>
>> lsof -p<pid> | grep deleted | wc -l gives a stable number (around 20,
>> growing and after decreasing : cleanup of gc or os... whatever...).
>>
>> No file in /tmp.
>>
>> I wonder if the files I found were files left further to the few "hard-stop"
>> I made during last week.
>> Currently, to stop James, I use kill -9<pid>
>>
>> Is there another cleaner way to do this ?
>> I saw shutdown command ("kills the current JVM (convenient when James is
>> run as a daemon)") in the remotemanager.
>> Is it different from kill -9 ? Should I use this ?
>>
>> Tks,
>>
>> Eric
>>
>>
>>
>> On 04/24/2010 04:23 PM, Norman Maurer wrote:
>>
>>> Hi Eric,
>>>
>>> just downloading it..
>>>
>>> About the tread safety. I don't think so. I'm still wonder if the
>>> exception was only thrown because of the OOM or if its the cause of
>>> the OOM. Hopefully I will have a better view of the whole story once I
>>> looked at the heap dump...
>>>
>>> Bye,
>>> Norman
>>>
>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>
>>>
>>>> Hi Norman,
>>>>
>>>> You can download it from
>>>> http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip
>>>>
>>>>
>>>> "This error seems to get thrown when you write to a closed stream.."
>>>> Could it be a thread-safety problem ? (on thread using the session/state
>>>> of
>>>> another one ?)
>>>>
>>>> I will also monitor with lsof -p<pid> | grep deleted | wc -l
>>>>
>>>> As config change, I activated a few days ago the ValidRcptHandler
>>>> fastfail
>>>> handler.
>>>>
>>>> Tks,
>>>>
>>>> Eric
>>>>
>>>>
>>>> On 04/24/2010 03:13 PM, Norman Maurer wrote:
>>>>
>>>>
>>>>> Could you upload the heap dump too ?
>>>>>
>>>>>
>>>>> 2010/4/24, Norman Maurer<no...@googlemail.com>:
>>>>>
>>>>>
>>>>>
>>>>>> Hi Eric,
>>>>>>
>>>>>> thx for the report. Comments are inline...
>>>>>>
>>>>>>
>>>>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I just got oom with netty impl.
>>>>>>> See http://apache.u-mangate.com/james/oom/20090424.png
>>>>>>> (http://apache.u-mangate.com/james/oom/index.html for history).
>>>>>>>
>>>>>>> Everything was fine with the trunk of last week that was really
>>>>>>> stable.
>>>>>>> I redeployed 3 days ago a new trunk version after a stop/start that
>>>>>>> showed
>>>>>>> ClassCastException of ImapMessage (ClassCastException was solved with
>>>>>>> the
>>>>>>> redeploy)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> The ClassCastException should be fixed since yesterday. This was
>>>>>> caused by the EntityManager got confused by entities..
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Short infos about the today oom:
>>>>>>> - The logs had IOException "Bad file descriptor" (see copy of the
>>>>>>> exception
>>>>>>> after this mail)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> This error seems to get thrown when you write to a closed stream...
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> - The begining time of these exceptions in the log is 00:10, and there
>>>>>>> are
>>>>>>> numerous ones during 1 minute.
>>>>>>> - The heap dump automatically generated via
>>>>>>> -XX:+HeapDumpOnOutOfMemoryError
>>>>>>> -XX:HeapDumpPath=ccured on 00:10, exactly on the same time the
>>>>>>> IOException "Bad file descriptor" began.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Ok..
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> - I forgot to lsof to see the open files... but when if the error came
>>>>>>> from
>>>>>>> too many open files, the received exception in the log would have been
>>>>>>> "java.io.IOException: too many open files".
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> right..
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> - Seems like an infinite loop as Norman anticipated, but now
>>>>>>> stacktraces
>>>>>>> from logs seem to confirm this.
>>>>>>> - Looking at line 150 of
>>>>>>>
>>>>>>>
>>>>>>> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=up
>>>>>>> , out.write(line) seem to be the place that triggers the exception,
>>>>>>> out
>>>>>>> variable being =tputStream)
>>>>>>> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Good catch will see how this could happen..
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> - The server filesystem seems sane (new hardware, running with enough
>>>>>>> privilege, other applications running on the same machine without any
>>>>>>> problem). I don't see why I have a "Bad file descriptor"
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> See above..
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> There are file in tmp with creation date that span 1 week (so not
>>>>>>> produced
>>>>>>> on 00:10):
>>>>>>> - A few messagestream*.ms
>>>>>>> - More rewindable*.tmp
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Will investigate ..
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> I restarted james.
>>>>>>> It still showed some issues (one account via pop3 gave errors in my
>>>>>>> client).
>>>>>>> oouch...
>>>>>>>
>>>>>>> I updated from trunk, redeployed, copied the var directory to migrate
>>>>>>> the
>>>>>>> account/domain/mail database, and restarted.
>>>>>>> Hopefully, now everything is working as before.
>>>>>>>
>>>>>>> So, we've got now a useful stacktrace that clearly shows where the oom
>>>>>>> "trigger" could be.
>>>>>>> I'm pretty sure that jconsole would have shown the same behaviour as
>>>>>>> http://apache.u-mangate.com/james/oom/oom.png
>>>>>>>
>>>>>>> Possible cause on the SMTP channel:
>>>>>>> - A weird behaviour of the smtp clients (hacking or respectfulness
>>>>>>> protocols
>>>>>>> clients)
>>>>>>> - A design in james causing infinite loop: thread clash (a thread
>>>>>>> accessing
>>>>>>> the same file as another thread?,...)
>>>>>>> - Additional James config (spamassassin or whatever) that leads to
>>>>>>> such
>>>>>>> behaviour
>>>>>>> - Hardware problem
>>>>>>>
>>>>>>> Currently, I run with default config + jdbcdomainlist.
>>>>>>> I except to receive another oom one of these days, unless the oom
>>>>>>> comes
>>>>>>> from
>>>>>>> an unactivated config (spamassassin or whatever).
>>>>>>>
>>>>>>> Tks,
>>>>>>>
>>>>>>> Eric
>>>>>>>
>>>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>>>>> processing DATA.
>>>>>>> java.io.IOException: Bad file descriptor
>>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
>>>>>>> at org.jboss.netty.channel.Channels.close(Channels.java:1065)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>>>>> processing DATA.
>>>>>>> java.io.IOException: Bad file descriptor
>>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>>> ...
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Bye,
>>>>>> Norman
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OOM with Netty
Posted by Norman Maurer <no...@googlemail.com>.
Hi Eric,
doesn't CTRL-C work in the terminal you started james in ? And you
should not use kill-9 <pid> because it will not let JAMES cleanup
stuff, if you want to use kill <pid> ..
Bye,
Norman
2010/4/24 Eric Charles <er...@u-mangate.com>:
> Hi Norman,
>
> lsof -p <pid> | grep deleted | wc -l gives a stable number (around 20,
> growing and after decreasing : cleanup of gc or os... whatever...).
>
> No file in /tmp.
>
> I wonder if the files I found were files left further to the few "hard-stop"
> I made during last week.
> Currently, to stop James, I use kill -9 <pid>
>
> Is there another cleaner way to do this ?
> I saw shutdown command ("kills the current JVM (convenient when James is
> run as a daemon)") in the remotemanager.
> Is it different from kill -9 ? Should I use this ?
>
> Tks,
>
> Eric
>
>
>
> On 04/24/2010 04:23 PM, Norman Maurer wrote:
>>
>> Hi Eric,
>>
>> just downloading it..
>>
>> About the tread safety. I don't think so. I'm still wonder if the
>> exception was only thrown because of the OOM or if its the cause of
>> the OOM. Hopefully I will have a better view of the whole story once I
>> looked at the heap dump...
>>
>> Bye,
>> Norman
>>
>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>
>>>
>>> Hi Norman,
>>>
>>> You can download it from
>>> http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip
>>>
>>>
>>> "This error seems to get thrown when you write to a closed stream.."
>>> Could it be a thread-safety problem ? (on thread using the session/state
>>> of
>>> another one ?)
>>>
>>> I will also monitor with lsof -p<pid> | grep deleted | wc -l
>>>
>>> As config change, I activated a few days ago the ValidRcptHandler
>>> fastfail
>>> handler.
>>>
>>> Tks,
>>>
>>> Eric
>>>
>>>
>>> On 04/24/2010 03:13 PM, Norman Maurer wrote:
>>>
>>>>
>>>> Could you upload the heap dump too ?
>>>>
>>>>
>>>> 2010/4/24, Norman Maurer<no...@googlemail.com>:
>>>>
>>>>
>>>>>
>>>>> Hi Eric,
>>>>>
>>>>> thx for the report. Comments are inline...
>>>>>
>>>>>
>>>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>>>
>>>>>
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I just got oom with netty impl.
>>>>>> See http://apache.u-mangate.com/james/oom/20090424.png
>>>>>> (http://apache.u-mangate.com/james/oom/index.html for history).
>>>>>>
>>>>>> Everything was fine with the trunk of last week that was really
>>>>>> stable.
>>>>>> I redeployed 3 days ago a new trunk version after a stop/start that
>>>>>> showed
>>>>>> ClassCastException of ImapMessage (ClassCastException was solved with
>>>>>> the
>>>>>> redeploy)
>>>>>>
>>>>>>
>>>>>
>>>>> The ClassCastException should be fixed since yesterday. This was
>>>>> caused by the EntityManager got confused by entities..
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Short infos about the today oom:
>>>>>> - The logs had IOException "Bad file descriptor" (see copy of the
>>>>>> exception
>>>>>> after this mail)
>>>>>>
>>>>>>
>>>>>
>>>>> This error seems to get thrown when you write to a closed stream...
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> - The begining time of these exceptions in the log is 00:10, and there
>>>>>> are
>>>>>> numerous ones during 1 minute.
>>>>>> - The heap dump automatically generated via
>>>>>> -XX:+HeapDumpOnOutOfMemoryError
>>>>>> -XX:HeapDumpPath= occured on 00:10, exactly on the same time the
>>>>>> IOException "Bad file descriptor" began.
>>>>>>
>>>>>>
>>>>>
>>>>> Ok..
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> - I forgot to lsof to see the open files... but when if the error came
>>>>>> from
>>>>>> too many open files, the received exception in the log would have been
>>>>>> "java.io.IOException: too many open files".
>>>>>>
>>>>>>
>>>>>
>>>>> right..
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> - Seems like an infinite loop as Norman anticipated, but now
>>>>>> stacktraces
>>>>>> from logs seem to confirm this.
>>>>>> - Looking at line 150 of
>>>>>>
>>>>>>
>>>>>> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=rkup
>>>>>> , out.write(line) seem to be the place that triggers the exception,
>>>>>> out
>>>>>> variable being =OutputStream)
>>>>>> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
>>>>>>
>>>>>>
>>>>>
>>>>> Good catch will see how this could happen..
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> - The server filesystem seems sane (new hardware, running with enough
>>>>>> privilege, other applications running on the same machine without any
>>>>>> problem). I don't see why I have a "Bad file descriptor"
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> See above..
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> There are file in tmp with creation date that span 1 week (so not
>>>>>> produced
>>>>>> on 00:10):
>>>>>> - A few messagestream*.ms
>>>>>> - More rewindable*.tmp
>>>>>>
>>>>>>
>>>>>
>>>>> Will investigate ..
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> I restarted james.
>>>>>> It still showed some issues (one account via pop3 gave errors in my
>>>>>> client).
>>>>>> oouch...
>>>>>>
>>>>>> I updated from trunk, redeployed, copied the var directory to migrate
>>>>>> the
>>>>>> account/domain/mail database, and restarted.
>>>>>> Hopefully, now everything is working as before.
>>>>>>
>>>>>> So, we've got now a useful stacktrace that clearly shows where the oom
>>>>>> "trigger" could be.
>>>>>> I'm pretty sure that jconsole would have shown the same behaviour as
>>>>>> http://apache.u-mangate.com/james/oom/oom.png
>>>>>>
>>>>>> Possible cause on the SMTP channel:
>>>>>> - A weird behaviour of the smtp clients (hacking or respectfulness
>>>>>> protocols
>>>>>> clients)
>>>>>> - A design in james causing infinite loop: thread clash (a thread
>>>>>> accessing
>>>>>> the same file as another thread?,...)
>>>>>> - Additional James config (spamassassin or whatever) that leads to
>>>>>> such
>>>>>> behaviour
>>>>>> - Hardware problem
>>>>>>
>>>>>> Currently, I run with default config + jdbcdomainlist.
>>>>>> I except to receive another oom one of these days, unless the oom
>>>>>> comes
>>>>>> from
>>>>>> an unactivated config (spamassassin or whatever).
>>>>>>
>>>>>> Tks,
>>>>>>
>>>>>> Eric
>>>>>>
>>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>>>> processing DATA.
>>>>>> java.io.IOException: Bad file descriptor
>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
>>>>>> at org.jboss.netty.channel.Channels.close(Channels.java:1065)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>>>> processing DATA.
>>>>>> java.io.IOException: Bad file descriptor
>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>>> ...
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> Bye,
>>>>> Norman
>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OOM with Netty
Posted by Eric Charles <er...@u-mangate.com>.
Hi Norman,
lsof -p <pid> | grep deleted | wc -l gives a stable number (around 20,
growing and after decreasing : cleanup of gc or os... whatever...).
No file in /tmp.
I wonder if the files I found were files left further to the few
"hard-stop" I made during last week.
Currently, to stop James, I use kill -9 <pid>
Is there another cleaner way to do this ?
I saw shutdown command ("kills the current JVM (convenient when James
is run as a daemon)") in the remotemanager.
Is it different from kill -9 ? Should I use this ?
Tks,
Eric
On 04/24/2010 04:23 PM, Norman Maurer wrote:
> Hi Eric,
>
> just downloading it..
>
> About the tread safety. I don't think so. I'm still wonder if the
> exception was only thrown because of the OOM or if its the cause of
> the OOM. Hopefully I will have a better view of the whole story once I
> looked at the heap dump...
>
> Bye,
> Norman
>
> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>
>> Hi Norman,
>>
>> You can download it from
>> http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip
>>
>>
>> "This error seems to get thrown when you write to a closed stream.."
>> Could it be a thread-safety problem ? (on thread using the session/state of
>> another one ?)
>>
>> I will also monitor with lsof -p<pid> | grep deleted | wc -l
>>
>> As config change, I activated a few days ago the ValidRcptHandler fastfail
>> handler.
>>
>> Tks,
>>
>> Eric
>>
>>
>> On 04/24/2010 03:13 PM, Norman Maurer wrote:
>>
>>> Could you upload the heap dump too ?
>>>
>>>
>>> 2010/4/24, Norman Maurer<no...@googlemail.com>:
>>>
>>>
>>>> Hi Eric,
>>>>
>>>> thx for the report. Comments are inline...
>>>>
>>>>
>>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>>
>>>>
>>>>> Hi All,
>>>>>
>>>>> I just got oom with netty impl.
>>>>> See http://apache.u-mangate.com/james/oom/20090424.png
>>>>> (http://apache.u-mangate.com/james/oom/index.html for history).
>>>>>
>>>>> Everything was fine with the trunk of last week that was really stable.
>>>>> I redeployed 3 days ago a new trunk version after a stop/start that
>>>>> showed
>>>>> ClassCastException of ImapMessage (ClassCastException was solved with
>>>>> the
>>>>> redeploy)
>>>>>
>>>>>
>>>> The ClassCastException should be fixed since yesterday. This was
>>>> caused by the EntityManager got confused by entities..
>>>>
>>>>
>>>>
>>>>
>>>>> Short infos about the today oom:
>>>>> - The logs had IOException "Bad file descriptor" (see copy of the
>>>>> exception
>>>>> after this mail)
>>>>>
>>>>>
>>>> This error seems to get thrown when you write to a closed stream...
>>>>
>>>>
>>>>
>>>>> - The begining time of these exceptions in the log is 00:10, and there
>>>>> are
>>>>> numerous ones during 1 minute.
>>>>> - The heap dump automatically generated via
>>>>> -XX:+HeapDumpOnOutOfMemoryError
>>>>> -XX:HeapDumpPath= occured on 00:10, exactly on the same time the
>>>>> IOException "Bad file descriptor" began.
>>>>>
>>>>>
>>>> Ok..
>>>>
>>>>
>>>>
>>>>> - I forgot to lsof to see the open files... but when if the error came
>>>>> from
>>>>> too many open files, the received exception in the log would have been
>>>>> "java.io.IOException: too many open files".
>>>>>
>>>>>
>>>> right..
>>>>
>>>>
>>>>
>>>>> - Seems like an infinite loop as Norman anticipated, but now stacktraces
>>>>> from logs seem to confirm this.
>>>>> - Looking at line 150 of
>>>>>
>>>>> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=rkup
>>>>> , out.write(line) seem to be the place that triggers the exception, out
>>>>> variable being =OutputStream)
>>>>> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
>>>>>
>>>>>
>>>> Good catch will see how this could happen..
>>>>
>>>>
>>>>
>>>>> - The server filesystem seems sane (new hardware, running with enough
>>>>> privilege, other applications running on the same machine without any
>>>>> problem). I don't see why I have a "Bad file descriptor"
>>>>>
>>>>>
>>>>>
>>>> See above..
>>>>
>>>>
>>>>
>>>>> There are file in tmp with creation date that span 1 week (so not
>>>>> produced
>>>>> on 00:10):
>>>>> - A few messagestream*.ms
>>>>> - More rewindable*.tmp
>>>>>
>>>>>
>>>> Will investigate ..
>>>>
>>>>
>>>>
>>>>> I restarted james.
>>>>> It still showed some issues (one account via pop3 gave errors in my
>>>>> client).
>>>>> oouch...
>>>>>
>>>>> I updated from trunk, redeployed, copied the var directory to migrate
>>>>> the
>>>>> account/domain/mail database, and restarted.
>>>>> Hopefully, now everything is working as before.
>>>>>
>>>>> So, we've got now a useful stacktrace that clearly shows where the oom
>>>>> "trigger" could be.
>>>>> I'm pretty sure that jconsole would have shown the same behaviour as
>>>>> http://apache.u-mangate.com/james/oom/oom.png
>>>>>
>>>>> Possible cause on the SMTP channel:
>>>>> - A weird behaviour of the smtp clients (hacking or respectfulness
>>>>> protocols
>>>>> clients)
>>>>> - A design in james causing infinite loop: thread clash (a thread
>>>>> accessing
>>>>> the same file as another thread?,...)
>>>>> - Additional James config (spamassassin or whatever) that leads to such
>>>>> behaviour
>>>>> - Hardware problem
>>>>>
>>>>> Currently, I run with default config + jdbcdomainlist.
>>>>> I except to receive another oom one of these days, unless the oom comes
>>>>> from
>>>>> an unactivated config (spamassassin or whatever).
>>>>>
>>>>> Tks,
>>>>>
>>>>> Eric
>>>>>
>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>>> processing DATA.
>>>>> java.io.IOException: Bad file descriptor
>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>> at
>>>>>
>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>> at
>>>>>
>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
>>>>> at org.jboss.netty.channel.Channels.close(Channels.java:1065)
>>>>> at
>>>>> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
>>>>> at
>>>>>
>>>>> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>> at
>>>>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>>> at
>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
>>>>> at
>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>>> at
>>>>>
>>>>> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>>> processing DATA.
>>>>> java.io.IOException: Bad file descriptor
>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>>> at
>>>>>
>>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>>> at
>>>>>
>>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>>> at
>>>>>
>>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>>> at
>>>>>
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>>> at
>>>>>
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>>> ...
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>> Bye,
>>>> Norman
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OOM with Netty
Posted by Norman Maurer <no...@googlemail.com>.
Hi Eric,
just downloading it..
About the tread safety. I don't think so. I'm still wonder if the
exception was only thrown because of the OOM or if its the cause of
the OOM. Hopefully I will have a better view of the whole story once I
looked at the heap dump...
Bye,
Norman
2010/4/24 Eric Charles <er...@u-mangate.com>:
> Hi Norman,
>
> You can download it from
> http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip
>
>
> "This error seems to get thrown when you write to a closed stream.."
> Could it be a thread-safety problem ? (on thread using the session/state of
> another one ?)
>
> I will also monitor with lsof -p <pid> | grep deleted | wc -l
>
> As config change, I activated a few days ago the ValidRcptHandler fastfail
> handler.
>
> Tks,
>
> Eric
>
>
> On 04/24/2010 03:13 PM, Norman Maurer wrote:
>>
>> Could you upload the heap dump too ?
>>
>>
>> 2010/4/24, Norman Maurer<no...@googlemail.com>:
>>
>>>
>>> Hi Eric,
>>>
>>> thx for the report. Comments are inline...
>>>
>>>
>>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>>
>>>>
>>>> Hi All,
>>>>
>>>> I just got oom with netty impl.
>>>> See http://apache.u-mangate.com/james/oom/20090424.png
>>>> (http://apache.u-mangate.com/james/oom/index.html for history).
>>>>
>>>> Everything was fine with the trunk of last week that was really stable.
>>>> I redeployed 3 days ago a new trunk version after a stop/start that
>>>> showed
>>>> ClassCastException of ImapMessage (ClassCastException was solved with
>>>> the
>>>> redeploy)
>>>>
>>>
>>> The ClassCastException should be fixed since yesterday. This was
>>> caused by the EntityManager got confused by entities..
>>>
>>>
>>>
>>>>
>>>> Short infos about the today oom:
>>>> - The logs had IOException "Bad file descriptor" (see copy of the
>>>> exception
>>>> after this mail)
>>>>
>>>
>>> This error seems to get thrown when you write to a closed stream...
>>>
>>>
>>>>
>>>> - The begining time of these exceptions in the log is 00:10, and there
>>>> are
>>>> numerous ones during 1 minute.
>>>> - The heap dump automatically generated via
>>>> -XX:+HeapDumpOnOutOfMemoryError
>>>> -XX:HeapDumpPath=/ occured on 00:10, exactly on the same time the
>>>> IOException "Bad file descriptor" began.
>>>>
>>>
>>> Ok..
>>>
>>>
>>>>
>>>> - I forgot to lsof to see the open files... but when if the error came
>>>> from
>>>> too many open files, the received exception in the log would have been
>>>> "java.io.IOException: too many open files".
>>>>
>>>
>>> right..
>>>
>>>
>>>>
>>>> - Seems like an infinite loop as Norman anticipated, but now stacktraces
>>>> from logs seem to confirm this.
>>>> - Looking at line 150 of
>>>>
>>>> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=markup
>>>> , out.write(line) seem to be the place that triggers the exception, out
>>>> variable being = (OutputStream)
>>>> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
>>>>
>>>
>>> Good catch will see how this could happen..
>>>
>>>
>>>>
>>>> - The server filesystem seems sane (new hardware, running with enough
>>>> privilege, other applications running on the same machine without any
>>>> problem). I don't see why I have a "Bad file descriptor"
>>>>
>>>>
>>>
>>> See above..
>>>
>>>
>>>>
>>>> There are file in tmp with creation date that span 1 week (so not
>>>> produced
>>>> on 00:10):
>>>> - A few messagestream*.ms
>>>> - More rewindable*.tmp
>>>>
>>>
>>> Will investigate ..
>>>
>>>
>>>>
>>>> I restarted james.
>>>> It still showed some issues (one account via pop3 gave errors in my
>>>> client).
>>>> oouch...
>>>>
>>>> I updated from trunk, redeployed, copied the var directory to migrate
>>>> the
>>>> account/domain/mail database, and restarted.
>>>> Hopefully, now everything is working as before.
>>>>
>>>> So, we've got now a useful stacktrace that clearly shows where the oom
>>>> "trigger" could be.
>>>> I'm pretty sure that jconsole would have shown the same behaviour as
>>>> http://apache.u-mangate.com/james/oom/oom.png
>>>>
>>>> Possible cause on the SMTP channel:
>>>> - A weird behaviour of the smtp clients (hacking or respectfulness
>>>> protocols
>>>> clients)
>>>> - A design in james causing infinite loop: thread clash (a thread
>>>> accessing
>>>> the same file as another thread?,...)
>>>> - Additional James config (spamassassin or whatever) that leads to such
>>>> behaviour
>>>> - Hardware problem
>>>>
>>>> Currently, I run with default config + jdbcdomainlist.
>>>> I except to receive another oom one of these days, unless the oom comes
>>>> from
>>>> an unactivated config (spamassassin or whatever).
>>>>
>>>> Tks,
>>>>
>>>> Eric
>>>>
>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>> processing DATA.
>>>> java.io.IOException: Bad file descriptor
>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>> at
>>>>
>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>> at
>>>>
>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>> at
>>>>
>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>> at
>>>>
>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>> at
>>>>
>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>> at
>>>>
>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>> at
>>>>
>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>> at
>>>>
>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>> at
>>>>
>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>> at
>>>>
>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>>
>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>>
>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>> at
>>>>
>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>> at
>>>>
>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>> at
>>>>
>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>> at
>>>>
>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>> at
>>>>
>>>> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
>>>> at
>>>>
>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
>>>> at
>>>>
>>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
>>>> at
>>>>
>>>> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
>>>> at
>>>>
>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
>>>> at org.jboss.netty.channel.Channels.close(Channels.java:1065)
>>>> at
>>>> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
>>>> at
>>>>
>>>> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>>
>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>>
>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>> at
>>>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
>>>> at
>>>>
>>>> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>> at
>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
>>>> at
>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>>>> at
>>>>
>>>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
>>>> at
>>>>
>>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>> at
>>>>
>>>> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>>> processing DATA.
>>>> java.io.IOException: Bad file descriptor
>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>> at
>>>>
>>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>> at
>>>>
>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>> at
>>>>
>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>> at
>>>>
>>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>> at
>>>>
>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>> at
>>>>
>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>> at
>>>>
>>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>> at
>>>>
>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>> at
>>>>
>>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>> at
>>>>
>>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>>
>>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>>
>>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>> at
>>>>
>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>> at
>>>>
>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>> at
>>>>
>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>> at
>>>>
>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>> at
>>>>
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>> at
>>>>
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>> ...
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>
>>> Bye,
>>> Norman
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OOM with Netty
Posted by Eric Charles <er...@u-mangate.com>.
Hi Norman,
You can download it from
http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip
"This error seems to get thrown when you write to a closed stream.."
Could it be a thread-safety problem ? (on thread using the session/state
of another one ?)
I will also monitor with lsof -p <pid> | grep deleted | wc -l
As config change, I activated a few days ago the ValidRcptHandler
fastfail handler.
Tks,
Eric
On 04/24/2010 03:13 PM, Norman Maurer wrote:
> Could you upload the heap dump too ?
>
>
> 2010/4/24, Norman Maurer<no...@googlemail.com>:
>
>> Hi Eric,
>>
>> thx for the report. Comments are inline...
>>
>>
>> 2010/4/24 Eric Charles<er...@u-mangate.com>:
>>
>>> Hi All,
>>>
>>> I just got oom with netty impl.
>>> See http://apache.u-mangate.com/james/oom/20090424.png
>>> (http://apache.u-mangate.com/james/oom/index.html for history).
>>>
>>> Everything was fine with the trunk of last week that was really stable.
>>> I redeployed 3 days ago a new trunk version after a stop/start that
>>> showed
>>> ClassCastException of ImapMessage (ClassCastException was solved with the
>>> redeploy)
>>>
>> The ClassCastException should be fixed since yesterday. This was
>> caused by the EntityManager got confused by entities..
>>
>>
>>
>>> Short infos about the today oom:
>>> - The logs had IOException "Bad file descriptor" (see copy of the
>>> exception
>>> after this mail)
>>>
>> This error seems to get thrown when you write to a closed stream...
>>
>>
>>> - The begining time of these exceptions in the log is 00:10, and there
>>> are
>>> numerous ones during 1 minute.
>>> - The heap dump automatically generated via
>>> -XX:+HeapDumpOnOutOfMemoryError
>>> -XX:HeapDumpPath=/ occured on 00:10, exactly on the same time the
>>> IOException "Bad file descriptor" began.
>>>
>> Ok..
>>
>>
>>> - I forgot to lsof to see the open files... but when if the error came
>>> from
>>> too many open files, the received exception in the log would have been
>>> "java.io.IOException: too many open files".
>>>
>> right..
>>
>>
>>> - Seems like an infinite loop as Norman anticipated, but now stacktraces
>>> from logs seem to confirm this.
>>> - Looking at line 150 of
>>> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=markup
>>> , out.write(line) seem to be the place that triggers the exception, out
>>> variable being = (OutputStream)
>>> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
>>>
>> Good catch will see how this could happen..
>>
>>
>>> - The server filesystem seems sane (new hardware, running with enough
>>> privilege, other applications running on the same machine without any
>>> problem). I don't see why I have a "Bad file descriptor"
>>>
>>>
>> See above..
>>
>>
>>> There are file in tmp with creation date that span 1 week (so not
>>> produced
>>> on 00:10):
>>> - A few messagestream*.ms
>>> - More rewindable*.tmp
>>>
>> Will investigate ..
>>
>>
>>> I restarted james.
>>> It still showed some issues (one account via pop3 gave errors in my
>>> client).
>>> oouch...
>>>
>>> I updated from trunk, redeployed, copied the var directory to migrate the
>>> account/domain/mail database, and restarted.
>>> Hopefully, now everything is working as before.
>>>
>>> So, we've got now a useful stacktrace that clearly shows where the oom
>>> "trigger" could be.
>>> I'm pretty sure that jconsole would have shown the same behaviour as
>>> http://apache.u-mangate.com/james/oom/oom.png
>>>
>>> Possible cause on the SMTP channel:
>>> - A weird behaviour of the smtp clients (hacking or respectfulness
>>> protocols
>>> clients)
>>> - A design in james causing infinite loop: thread clash (a thread
>>> accessing
>>> the same file as another thread?,...)
>>> - Additional James config (spamassassin or whatever) that leads to such
>>> behaviour
>>> - Hardware problem
>>>
>>> Currently, I run with default config + jdbcdomainlist.
>>> I except to receive another oom one of these days, unless the oom comes
>>> from
>>> an unactivated config (spamassassin or whatever).
>>>
>>> Tks,
>>>
>>> Eric
>>>
>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>> processing DATA.
>>> java.io.IOException: Bad file descriptor
>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>> at
>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>> at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>> at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>> at
>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>> at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>> at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>> at
>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>> at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>> at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>> at
>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>> at
>>> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
>>> at
>>> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
>>> at
>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
>>> at org.jboss.netty.channel.Channels.close(Channels.java:1065)
>>> at
>>> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
>>> at
>>> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>> at
>>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
>>> at
>>> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
>>> at
>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> at
>>> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:619)
>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>> processing DATA.
>>> java.io.IOException: Bad file descriptor
>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>> at
>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>> at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>> at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>> at
>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>> at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>> at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>> at
>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>> at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>> at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>> at
>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>> ...
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>> Bye,
>> Norman
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OOM with Netty
Posted by Norman Maurer <no...@apache.org>.
Could you upload the heap dump too ?
2010/4/24, Norman Maurer <no...@googlemail.com>:
> Hi Eric,
>
> thx for the report. Comments are inline...
>
>
> 2010/4/24 Eric Charles <er...@u-mangate.com>:
>> Hi All,
>>
>> I just got oom with netty impl.
>> See http://apache.u-mangate.com/james/oom/20090424.png
>> (http://apache.u-mangate.com/james/oom/index.html for history).
>>
>> Everything was fine with the trunk of last week that was really stable.
>> I redeployed 3 days ago a new trunk version after a stop/start that
>> showed
>> ClassCastException of ImapMessage (ClassCastException was solved with the
>> redeploy)
>
> The ClassCastException should be fixed since yesterday. This was
> caused by the EntityManager got confused by entities..
>
>
>>
>> Short infos about the today oom:
>> - The logs had IOException "Bad file descriptor" (see copy of the
>> exception
>> after this mail)
>
> This error seems to get thrown when you write to a closed stream...
>
>> - The begining time of these exceptions in the log is 00:10, and there
>> are
>> numerous ones during 1 minute.
>> - The heap dump automatically generated via
>> -XX:+HeapDumpOnOutOfMemoryError
>> -XX:HeapDumpPath=/ occured on 00:10, exactly on the same time the
>> IOException "Bad file descriptor" began.
>
> Ok..
>
>> - I forgot to lsof to see the open files... but when if the error came
>> from
>> too many open files, the received exception in the log would have been
>> "java.io.IOException: too many open files".
>
> right..
>
>> - Seems like an infinite loop as Norman anticipated, but now stacktraces
>> from logs seem to confirm this.
>> - Looking at line 150 of
>> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=markup
>> , out.write(line) seem to be the place that triggers the exception, out
>> variable being = (OutputStream)
>> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
>
> Good catch will see how this could happen..
>
>> - The server filesystem seems sane (new hardware, running with enough
>> privilege, other applications running on the same machine without any
>> problem). I don't see why I have a "Bad file descriptor"
>>
>
> See above..
>
>> There are file in tmp with creation date that span 1 week (so not
>> produced
>> on 00:10):
>> - A few messagestream*.ms
>> - More rewindable*.tmp
>
> Will investigate ..
>
>>
>> I restarted james.
>> It still showed some issues (one account via pop3 gave errors in my
>> client).
>> oouch...
>>
>> I updated from trunk, redeployed, copied the var directory to migrate the
>> account/domain/mail database, and restarted.
>> Hopefully, now everything is working as before.
>>
>> So, we've got now a useful stacktrace that clearly shows where the oom
>> "trigger" could be.
>> I'm pretty sure that jconsole would have shown the same behaviour as
>> http://apache.u-mangate.com/james/oom/oom.png
>>
>> Possible cause on the SMTP channel:
>> - A weird behaviour of the smtp clients (hacking or respectfulness
>> protocols
>> clients)
>> - A design in james causing infinite loop: thread clash (a thread
>> accessing
>> the same file as another thread?,...)
>> - Additional James config (spamassassin or whatever) that leads to such
>> behaviour
>> - Hardware problem
>>
>> Currently, I run with default config + jdbcdomainlist.
>> I except to receive another oom one of these days, unless the oom comes
>> from
>> an unactivated config (spamassassin or whatever).
>>
>> Tks,
>>
>> Eric
>>
>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>> processing DATA.
>> java.io.IOException: Bad file descriptor
>> at java.io.FileOutputStream.writeBytes(Native Method)
>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>> at
>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>> at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>> at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>> at
>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>> at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>> at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>> at
>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>> at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>> at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>> at
>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>> at
>> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
>> at
>> org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
>> at
>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
>> at
>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
>> at
>> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
>> at
>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
>> at org.jboss.netty.channel.Channels.close(Channels.java:1065)
>> at
>> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
>> at
>> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>> at
>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
>> at
>> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>> at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
>> at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
>> at
>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>> at
>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
>> at
>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
>> at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at
>> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>> processing DATA.
>> java.io.IOException: Bad file descriptor
>> at java.io.FileOutputStream.writeBytes(Native Method)
>> at java.io.FileOutputStream.write(FileOutputStream.java:247)
>> at
>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>> at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>> at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>> at
>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>> at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>> at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>> at
>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>> at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>> at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>> at
>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>> ...
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
> Bye,
> Norman
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OOM with Netty
Posted by Norman Maurer <no...@googlemail.com>.
Hi Eric,
thx for the report. Comments are inline...
2010/4/24 Eric Charles <er...@u-mangate.com>:
> Hi All,
>
> I just got oom with netty impl.
> See http://apache.u-mangate.com/james/oom/20090424.png
> (http://apache.u-mangate.com/james/oom/index.html for history).
>
> Everything was fine with the trunk of last week that was really stable.
> I redeployed 3 days ago a new trunk version after a stop/start that showed
> ClassCastException of ImapMessage (ClassCastException was solved with the
> redeploy)
The ClassCastException should be fixed since yesterday. This was
caused by the EntityManager got confused by entities..
>
> Short infos about the today oom:
> - The logs had IOException "Bad file descriptor" (see copy of the exception
> after this mail)
This error seems to get thrown when you write to a closed stream...
> - The begining time of these exceptions in the log is 00:10, and there are
> numerous ones during 1 minute.
> - The heap dump automatically generated via -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/ occured on 00:10, exactly on the same time the
> IOException "Bad file descriptor" began.
Ok..
> - I forgot to lsof to see the open files... but when if the error came from
> too many open files, the received exception in the log would have been
> "java.io.IOException: too many open files".
right..
> - Seems like an infinite loop as Norman anticipated, but now stacktraces
> from logs seem to confirm this.
> - Looking at line 150 of
> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=markup
> , out.write(line) seem to be the place that triggers the exception, out
> variable being = (OutputStream)
> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
Good catch will see how this could happen..
> - The server filesystem seems sane (new hardware, running with enough
> privilege, other applications running on the same machine without any
> problem). I don't see why I have a "Bad file descriptor"
>
See above..
> There are file in tmp with creation date that span 1 week (so not produced
> on 00:10):
> - A few messagestream*.ms
> - More rewindable*.tmp
Will investigate ..
>
> I restarted james.
> It still showed some issues (one account via pop3 gave errors in my client).
> oouch...
>
> I updated from trunk, redeployed, copied the var directory to migrate the
> account/domain/mail database, and restarted.
> Hopefully, now everything is working as before.
>
> So, we've got now a useful stacktrace that clearly shows where the oom
> "trigger" could be.
> I'm pretty sure that jconsole would have shown the same behaviour as
> http://apache.u-mangate.com/james/oom/oom.png
>
> Possible cause on the SMTP channel:
> - A weird behaviour of the smtp clients (hacking or respectfulness protocols
> clients)
> - A design in james causing infinite loop: thread clash (a thread accessing
> the same file as another thread?,...)
> - Additional James config (spamassassin or whatever) that leads to such
> behaviour
> - Hardware problem
>
> Currently, I run with default config + jdbcdomainlist.
> I except to receive another oom one of these days, unless the oom comes from
> an unactivated config (spamassassin or whatever).
>
> Tks,
>
> Eric
>
> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
> processing DATA.
> java.io.IOException: Bad file descriptor
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:247)
> at
> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
> at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
> at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
> at
> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
> at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
> at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
> at
> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
> at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
> at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
> at
> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
> at
> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
> at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
> at
> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
> at org.jboss.netty.channel.Channels.close(Channels.java:1065)
> at
> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
> at
> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
> at
> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
> at
> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
> processing DATA.
> java.io.IOException: Bad file descriptor
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:247)
> at
> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
> at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
> at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
> at
> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
> at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
> at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
> at
> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
> at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
> at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
> at
> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
> ...
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
Bye,
Norman
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org