Expiring connection ActiveMQConnection due to "java.io.EOFException"

["sperling.alexander" <sp...@dakosy.de>]

Hello,

we have strange exceptions after some time with our ActiveMQ (version
5.11.1) installation. This ActiveMQ is used by several Apache Camel services
for communication to each other. 

The first warning we've got in our log was:



After that there are a lot of 



The log of activemq broker contains following line (there is a 2 hours gap
because of UTC setting):



We suppose, that there was a large amount of data inside a message of a
queue. Is there another reason for this exception and better, is there a
solution for that?



--
View this message in context: http://activemq.2283324.n4.nabble.com/Expiring-connection-ActiveMQConnection-due-to-java-io-EOFException-tp4702005.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Expiring connection ActiveMQConnection due to "java.io.EOFException"

[Tim Bain <tb...@alumni.duke.edu>]

It looks like the connection was closed because the inactivity monitor
didn't receive any data (keep-alive or otherwise) for longer than its
timeout.  The keep-alive packets are supposed to be sent every 1/3 of the
keep-alive interval if no other content is being sent.

That can happen if there is a network comms problem (disconnection, heavy
packet loss, network saturation, etc.) between the two ends of the
connection for > 2/3 of the interval.  It can happen if either end of the
connection has a full garbage collect that makes the program unresponsive
for more than 2/3 of the interval.  It can happen if thread contention or
OS resource contention are so severe that the thread doing the keep-alive
sending or the thread reading from the connection can't be scheduled for
more than 2/3 of the interval.

All of those are cases where it's expected that the inactivity monitor
would kick in, and if any of them occurred during that timeframe, then
that's probably the explanation.  Unfortunately, unless you've proactively
set up really good monitoring for those things, you probably won't be able
to figure out if any of them happened to you then.

With that being said, I don't trust the inactivity monitor code.  I've
personally observed sporadic disconnections due to the inactivity monitor
when we couldn't detect anything that could explain it.  The monitor has
been shown to completely fail to detect the problem on one side of the
connection when you pull a network cable (search the archives if you're
interested in the details).  And there are posts like yours in the mailing
list archives and on places like StackOverflow, indicating that you and I
aren't the only ones who have seen this behavior.

All of this observed behavior is a smell, and as a result my confidence in
the inactivity monitor's correct operation 100.0% of the time is very low.
With that being said, I haven't taken the time to look at the code to see
if there are any subtle problems there, so this is a smell and a feeling,
not proof that anything is wrong.  (Except for the cable-pull scenario;
that is a very obvious bug, which might or might not have been fixed since
the mailing list thread about it was written.)  If you have time, grab the
source code and look through that portion of it and see if you see anything
fishy...

Tim
On Sep 15, 2015 7:04 AM, "sperling.alexander" <sp...@dakosy.de> wrote:

> Hello,
>
> we have strange exceptions after some time with our ActiveMQ (version
> 5.11.1) installation. This ActiveMQ is used by several Apache Camel
> services
> for communication to each other.
>
> The first warning we've got in our log was:
>
>
>
> After that there are a lot of
>
>
>
> The log of activemq broker contains following line (there is a 2 hours gap
> because of UTC setting):
>
>
>
> We suppose, that there was a large amount of data inside a message of a
> queue. Is there another reason for this exception and better, is there a
> solution for that?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Expiring-connection-ActiveMQConnection-due-to-java-io-EOFException-tp4702005.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>