You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@curator.apache.org by Henrik Nordvik <he...@gmail.com> on 2014/01/14 16:19:55 UTC

ConnectionLossException logs a lot on retry

Hi,
I've hit an issue where I get an error message way too often. This
message repeats about 10 times every millisecond, which fills our
disks with logs.
This happened when zookeeper went down.

2014-01-14 00:00:04,464 ERROR [mework-0-EventThread]
o.a.c.f.imps.CuratorFrameworkImpl         - Background operation retry
gave up
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
~[zookeeper-3.4.5.jar:3.4.5-1392090]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:583)
[curator-framework-2.3.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:478)
[curator-framework-2.3.0.jar:na]
        at org.apache.curator.framework.imps.DeleteBuilderImpl$2.processResult(DeleteBuilderImpl.java:161)
[curator-framework-2.3.0.jar:na]
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:606)
[zookeeper-3.4.5.jar:3.4.5-1392090]
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[zookeeper-3.4.5.jar:3.4.5-1392090]
2014-01-14 00:00:04,464 ERROR [mework-0-EventThread]
o.a.c.f.imps.CuratorFrameworkImpl         - Background operation retry
gave up
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
~[zookeeper-3.4.5.jar:3.4.5-1392090]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:583)
[curator-framework-2.3.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:478)
[curator-framework-2.3.0.jar:na]
        at org.apache.curator.framework.imps.DeleteBuilderImpl$2.processResult(DeleteBuilderImpl.java:161)
[curator-framework-2.3.0.jar:na]
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:606)
[zookeeper-3.4.5.jar:3.4.5-1392090]
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[zookeeper-3.4.5.jar:3.4.5-1392090]
2014-01-14 00:00:04,465 ERROR [mework-0-EventThread]
o.a.c.f.imps.CuratorFrameworkImpl         - Background operation retry
gave up
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
~[zookeeper-3.4.5.jar:3.4.5-1392090]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:583)
[curator-framework-2.3.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:478)
[curator-framework-2.3.0.jar:na]
        at org.apache.curator.framework.imps.DeleteBuilderImpl$2.processResult(DeleteBuilderImpl.java:161)
[curator-framework-2.3.0.jar:na]
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:606)
[zookeeper-3.4.5.jar:3.4.5-1392090]
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[zookeeper-3.4.5.jar:3.4.5-1392090]

Is this an issue with zookeeper or curator?
How can I configure this so that it's not logging extreme amounts?

--
Henrik Nordvik

Re: ConnectionLossException logs a lot on retry

Posted by Henrik Nordvik <he...@gmail.com>.
Error is the highest logging level for slf4j, so by setting it to
Fatal effectively disables logging for that class.

I also see the "curator-dont-log-connection-problems" system property
I can set to disable this error (and some other) specifically.

Is this the right way of handling it? Is this hiding an underlying issue?

--
Henrik Nordvik


On Wed, Jan 15, 2014 at 8:49 AM, Evaristo José Camarero
<ev...@yahoo.es> wrote:
> Hi there,
>
> I have seen this behaviour in Curator 2.3.0 when ZK ensemble is down and at
> least this was not happening with Curator 2.1.0. I am not really sure if
> this is software bug.
>
> Curator relies on slf4j , so the logging configuration can be tune in the
> config file of the logging framwework you are  using.
>
> If you are using log4j and example is using the following line:
> log4j.logger.org.apache.curator = FATAL
> but then you will miss all the ERROR information
> You can also use finer grane using:
> log4.logger.o.a.c.f.imps.CuratorFrameworkImpl = FATAL
>
> Regards,
>
> Evaristo
>
>
>
> El Martes 14 de enero de 2014 16:20, Henrik Nordvik <he...@gmail.com>
> escribió:
> Hi,
> I've hit an issue where I get an error message way too often. This
> message repeats about 10 times every millisecond, which fills our
> disks with logs.
> This happened when zookeeper went down.
>
> 2014-01-14 00:00:04,464 ERROR [mework-0-EventThread]
> o.a.c.f.imps.CuratorFrameworkImpl        - Background operation retry
> gave up
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> ~[zookeeper-3.4.5.jar:3.4.5-1392090]
>         at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:583)
> [curator-framework-2.3.0.jar:na]
>         at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:478)
> [curator-framework-2.3.0.jar:na]
>         at
> org.apache.curator.framework.imps.DeleteBuilderImpl$2.processResult(DeleteBuilderImpl.java:161)
> [curator-framework-2.3.0.jar:na]
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:606)
> [zookeeper-3.4.5.jar:3.4.5-1392090]
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> [zookeeper-3.4.5.jar:3.4.5-1392090]
> 2014-01-14 00:00:04,464 ERROR [mework-0-EventThread]
> o.a.c.f.imps.CuratorFrameworkImpl        - Background operation retry
> gave up
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> ~[zookeeper-3.4.5.jar:3.4.5-1392090]
>         at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:583)
> [curator-framework-2.3.0.jar:na]
>         at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:478)
> [curator-framework-2.3.0.jar:na]
>         at
> org.apache.curator.framework.imps.DeleteBuilderImpl$2.processResult(DeleteBuilderImpl.java:161)
> [curator-framework-2.3.0.jar:na]
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:606)
> [zookeeper-3.4.5.jar:3.4.5-1392090]
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> [zookeeper-3.4.5.jar:3.4.5-1392090]
> 2014-01-14 00:00:04,465 ERROR [mework-0-EventThread]
> o.a.c.f.imps.CuratorFrameworkImpl        - Background operation retry
> gave up
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> ~[zookeeper-3.4.5.jar:3.4.5-1392090]
>         at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:583)
> [curator-framework-2.3.0.jar:na]
>         at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:478)
> [curator-framework-2.3.0.jar:na]
>         at
> org.apache.curator.framework.imps.DeleteBuilderImpl$2.processResult(DeleteBuilderImpl.java:161)
> [curator-framework-2.3.0.jar:na]
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:606)
> [zookeeper-3.4.5.jar:3.4.5-1392090]
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> [zookeeper-3.4.5.jar:3.4.5-1392090]
>
> Is this an issue with zookeeper or curator?
> How can I configure this so that it's not logging extreme amounts?
>
> --
> Henrik Nordvik
>
>

Re: ConnectionLossException logs a lot on retry

Posted by Evaristo José Camarero <ev...@yahoo.es>.
Hi there,

I have seen this behaviour in Curator 2.3.0 when ZK ensemble is down and at least this was not happening with Curator 2.1.0. I am not really sure if this is software bug.

Curator relies on slf4j , so the logging configuration can be tune in the config file of the logging framwework you are  using.

If you are using log4j and example is using the following line:
log4j.logger.org.apache.curator = FATAL

but then you will miss all the ERROR information
You can also use finer grane using:

log4.logger.o.a.c.f.imps.CuratorFrameworkImpl = FATAL


Regards,

Evaristo




El Martes 14 de enero de 2014 16:20, Henrik Nordvik <he...@gmail.com> escribió:
 
Hi,
I've hit an issue where I get an error message way too often. This
message repeats about 10 times every millisecond, which fills our
disks with logs.
This happened when zookeeper went down.

2014-01-14 00:00:04,464 ERROR [mework-0-EventThread]
o.a.c.f.imps.CuratorFrameworkImpl         - Background operation retry
gave up
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
~[zookeeper-3.4.5.jar:3.4.5-1392090]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:583)
[curator-framework-2.3.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:478)
[curator-framework-2.3.0.jar:na]
        at org.apache.curator.framework.imps.DeleteBuilderImpl$2.processResult(DeleteBuilderImpl.java:161)
[curator-framework-2.3.0.jar:na]
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:606)
[zookeeper-3.4.5.jar:3.4.5-1392090]
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[zookeeper-3.4.5.jar:3.4.5-1392090]
2014-01-14 00:00:04,464 ERROR [mework-0-EventThread]
o.a.c.f.imps.CuratorFrameworkImpl         - Background operation retry
gave up
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
~[zookeeper-3.4.5.jar:3.4.5-1392090]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:583)
[curator-framework-2.3.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:478)
[curator-framework-2.3.0.jar:na]
        at org.apache.curator.framework.imps.DeleteBuilderImpl$2.processResult(DeleteBuilderImpl.java:161)
[curator-framework-2.3.0.jar:na]
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:606)
[zookeeper-3.4.5.jar:3.4.5-1392090]
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[zookeeper-3.4.5.jar:3.4.5-1392090]
2014-01-14 00:00:04,465 ERROR [mework-0-EventThread]
o.a.c.f.imps.CuratorFrameworkImpl         - Background operation retry
gave up
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
~[zookeeper-3.4.5.jar:3.4.5-1392090]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:583)
[curator-framework-2.3.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:478)
[curator-framework-2.3.0.jar:na]
        at org.apache.curator.framework.imps.DeleteBuilderImpl$2.processResult(DeleteBuilderImpl.java:161)
[curator-framework-2.3.0.jar:na]
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:606)
[zookeeper-3.4.5.jar:3.4.5-1392090]
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[zookeeper-3.4.5.jar:3.4.5-1392090]

Is this an issue with zookeeper or curator?
How can I configure this so that it's not logging extreme amounts?

--
Henrik Nordvik