You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Jostein Elvaker Haande <je...@gmail.com> on 2018/07/16 11:04:00 UTC
Problem with ZK log files and snapshots
Hello,
I'm using ZooKeeper 3.4.12, and I'm facing an issue where the ZK log files
keep up eating disk space. I'm running this on several machines, and the
symptoms I'm seeing are the seem:
- I see a lot of log files
- Rarely, if ever any snapshot files
I started digging into it, and from
https://zookeeper.apache.org/doc/r3.4.12/zookeeperAdmin.html#sc_maintenance
one should use the ZK Java class org.apache.zookeeper.server.PurgeTxnLog
for this. However, seeing as I rarely see any snapshot files, PurgeTxnLog
never performs any cleanup.
Going deeper down, I can see from the documentation that a snapshot is
created every 100K transactions that are kept in the log files, so I was
interested in seeing how man transaction my logs kept. I then made a script
that used the Java class org.apache.zookeeper.server.LogFormatter which
outputs number of transactions on the same line, and I noticed that my ZK
instances saw a very low transaction count. For most of my instances, the
transaction count was below 20K.
Now I know that by setting snapCount to a value lower than the default
100K, it should generate snapshots more frequently, and ZK should then be
able to do this cleanup automatically which is controlled
by autopurge.snapRetainCount and autopurge.purgeInterval.
However, when setting a new value for snapCount, this does not seem to have
any effect on the existing transaction log files. So my question is, how do
I deal with the old transaction logs? How do I proceed to clean this away
safely?
--
Yours sincerely Jostein Elvaker Haande
"A free society is a society where it is safe to be unpopular"
- Adlai Stevenson
http://tolecnal.net -- tolecnal at tolecnal dot net
Re: Problem with ZK log files and snapshots
Posted by Jostein Elvaker Haande <je...@gmail.com>.
So I was able to schedule a maintenance window for one of the instances,
and performed the following:
- Performed a VM snapshot of the instance
- Stopped ZooKeeper
- Deleted all log and snapshot files
- Started ZooKeeper
What I noticed then, was that both ZooKeeper and SOLR was able to start.
However when I entered SOLR admin panla, it complained about not finding
the configuration files for the collections and shards. This was easily
fixed by uploading the configuration to ZK again, and performing a new
restart of the service. Once this was performed, everything was back to
working order.
I also observed that now that the old log files and snapshots were gone,
new log files and snapshots were finally created according to the values
defined by snapCount, which means I can finally rely on either using
PurgeTxnLog or have ZK do it automatically according to the values set
in autopurge.snapRetainCount
and autopurge.purgeInterval.
--
Yours sincerely Jostein Elvaker Haande
"A free society is a society where it is safe to be unpopular"
- Adlai Stevenson
http://tolecnal.net -- tolecnal at tolecnal dot net
Re: Problem with ZK log files and snapshots
Posted by Jostein Elvaker Haande <je...@gmail.com>.
On Wed, 18 Jul 2018 at 12:28, Norbert Kalmar <nk...@cloudera.com.invalid>
wrote:
> Hi,
>
> You should use PurgeTxnLog as per
> https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#sc_maintenance
> It needs 3 parameters: txn log dir, smapshot dir and number of snaps/log to
> keep (3 minimum!)
>
Hi,
I'm well aware of PurgeTxnLog (as per my initial post), however the problem
I'm facing is that there are not enough transactions in the existing log
files to generate a snapshot.
And seeing as no snapshots are created, PurgeTxnLog has nothing to clean
up. I've decreased the 'snapCount' on a few instances, however lowering
this count does nothing for the existing transaction logs it seems, as no
snapshots are created after a restart of ZooKeeper.
--
Yours sincerely Jostein Elvaker Haande
"A free society is a society where it is safe to be unpopular"
- Adlai Stevenson
http://tolecnal.net -- tolecnal at tolecnal dot net
Re: Problem with ZK log files and snapshots
Posted by Norbert Kalmar <nk...@cloudera.com.INVALID>.
Hi,
You should use PurgeTxnLog as per
https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#sc_maintenance
It needs 3 parameters: txn log dir, smapshot dir and number of snaps/log to
keep (3 minimum!)
Example:
java -cp
zookeeper.jar:lib/slf4j-api-1.6.1.jar:lib/slf4j-log4j12-1.6.1.jar:lib/log4j-1.2.15.jar:conf
org.apache.zookeeper.server.PurgeTxnLog /my/txn/log /my/snapshot/dir -n 3
A cronjob can handle the cleanup.
Or from 3.4.0 you can enavle it automatically: autopurge.snapRetainCount
and autopurge.purgeInterval
See
https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#sc_advancedConfiguration
Regards,
Norbert
On Wed, Jul 18, 2018 at 11:49 AM Jostein Elvaker Haande <je...@gmail.com>
wrote:
> Another thing I can't make heads or tails of, is whether or not it's safe
> to delete these transaction logs and snapshots manually.
>
> If yes, should this be done while ZooKeeper is stopped? Some of my
> instances are consuming quite a bit of disk space because of this issue,
> and to keep increasing disk space to "fix" the issue just isn't an option.
>
> --
> Yours sincerely Jostein Elvaker Haande
> "A free society is a society where it is safe to be unpopular"
> - Adlai Stevenson
>
> http://tolecnal.net -- tolecnal at tolecnal dot net
>
Re: Problem with ZK log files and snapshots
Posted by Jostein Elvaker Haande <je...@gmail.com>.
Another thing I can't make heads or tails of, is whether or not it's safe
to delete these transaction logs and snapshots manually.
If yes, should this be done while ZooKeeper is stopped? Some of my
instances are consuming quite a bit of disk space because of this issue,
and to keep increasing disk space to "fix" the issue just isn't an option.
--
Yours sincerely Jostein Elvaker Haande
"A free society is a society where it is safe to be unpopular"
- Adlai Stevenson
http://tolecnal.net -- tolecnal at tolecnal dot net
Re: Problem with ZK log files and snapshots
Posted by Baskar Duraikannu <ba...@outlook.com>.
If your transactions are short lived (something like you are taking a lock and then releasing it), you will find that “containers” holding these locks are not cleaned by Zookeeper. This in turn will make snapshot files big and will cause all sorts of issues. I found myself in that situation few months back. New version of zookeeper has container support which solves this issue. We had this issue and upgrading to 3.5.4-Beta and appropriate version of curator solved the issue.
Good luck.
> On Jul 16, 2018, at 7:04 AM, Jostein Elvaker Haande <je...@gmail.com> wrote:
>
> Hello,
>
> I'm using ZooKeeper 3.4.12, and I'm facing an issue where the ZK log files
> keep up eating disk space. I'm running this on several machines, and the
> symptoms I'm seeing are the seem:
>
> - I see a lot of log files
> - Rarely, if ever any snapshot files
>
> I started digging into it, and from
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fzookeeper.apache.org%2Fdoc%2Fr3.4.12%2FzookeeperAdmin.html%23sc_maintenance&data=02%7C01%7C%7Ca930ebc3267246f355f208d5eb0bddf9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636673358550132623&sdata=pBKODIXN1Yl5%2BL7zAtWvgoocWK20g8BAbFygPFpa648%3D&reserved=0
> one should use the ZK Java class org.apache.zookeeper.server.PurgeTxnLog
> for this. However, seeing as I rarely see any snapshot files, PurgeTxnLog
> never performs any cleanup.
>
> Going deeper down, I can see from the documentation that a snapshot is
> created every 100K transactions that are kept in the log files, so I was
> interested in seeing how man transaction my logs kept. I then made a script
> that used the Java class org.apache.zookeeper.server.LogFormatter which
> outputs number of transactions on the same line, and I noticed that my ZK
> instances saw a very low transaction count. For most of my instances, the
> transaction count was below 20K.
>
> Now I know that by setting snapCount to a value lower than the default
> 100K, it should generate snapshots more frequently, and ZK should then be
> able to do this cleanup automatically which is controlled
> by autopurge.snapRetainCount and autopurge.purgeInterval.
>
> However, when setting a new value for snapCount, this does not seem to have
> any effect on the existing transaction log files. So my question is, how do
> I deal with the old transaction logs? How do I proceed to clean this away
> safely?
>
> --
> Yours sincerely Jostein Elvaker Haande
> "A free society is a society where it is safe to be unpopular"
> - Adlai Stevenson
>
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftolecnal.net&data=02%7C01%7C%7Ca930ebc3267246f355f208d5eb0bddf9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636673358550132623&sdata=kUcc%2BHDxvoBG%2F7D3HQx2XxsPHfDoQmkRtXH6fVlN5To%3D&reserved=0 -- tolecnal at tolecnal dot net