You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Ryan Fox <ry...@mediabeacon.com> on 2013/10/30 16:05:43 UTC
auto-purge not purging
Hello,
I am running solr 4.1.0 with one shard and two replicas. One of my replica
nodes is running an embedded zookeeper instance. Per the administrator's
guide, I have the properties `autopurge.snapRetainCount` and
`autopurge.purgeInterval` set in zoo.cfg. Despite this, my transaction
logs/snapshots do not get auto-purged, even after the interval has expired
several times over.
Is this related to running zookeeper as an embedded process?
Thank you,
Ryan Fox
Re: auto-purge not purging
Posted by Ryan Fox <ry...@mediabeacon.com>.
Ah, sorry I wasn't clear. There's currently 3 log/snapshot pairs in that
dir, because I cleared it out recently. If I keep starting and stopping
solr/zookeeper, they'll continue to pile up.
$ ls -lah
drwxr-xr-x 14 ryanfox staff 476B Nov 4 17:25 .
drwxr-xr-x 4 ryanfox staff 136B Nov 4 09:30 ..
-rw-r--r-- 1 ryanfox staff 64M Oct 30 13:09 log.1
-rw-r--r-- 1 ryanfox staff 64M Nov 4 17:25 log.112
-rw-r--r-- 1 ryanfox staff 64M Nov 4 09:36 log.48
-rw-r--r-- 1 ryanfox staff 64M Nov 4 17:22 log.7b
-rw-r--r-- 1 ryanfox staff 64M Nov 4 17:22 log.ad
-rw-r--r-- 1 ryanfox staff 64M Nov 4 17:25 log.e1
-rw-r--r-- 1 ryanfox staff 296B Oct 30 12:53 snapshot.0
-rw-r--r-- 1 ryanfox staff 60K Nov 4 17:25 snapshot.111
-rw-r--r-- 1 ryanfox staff 60K Nov 4 09:34 snapshot.47
-rw-r--r-- 1 ryanfox staff 60K Nov 4 09:36 snapshot.7a
-rw-r--r-- 1 ryanfox staff 60K Nov 4 17:22 snapshot.ac
-rw-r--r-- 1 ryanfox staff 60K Nov 4 17:23 snapshot.e0
Thanks for the info on preAllocSize, that may prove useful as well.
Nothing showed up in the logs today despite running for ~8 hours.
Best,
Ryan
On Mon, Nov 4, 2013 at 5:13 PM, Patrick Hunt <ph...@apache.org> wrote:
> On Mon, Nov 4, 2013 at 7:46 AM, Ryan Fox <ry...@mediabeacon.com> wrote:
> >>
> >> What version of ZooKeeper are you running? (solr 4.1.0 is pretty old,
> >> 4.5.1 is out now, perhaps try upgrading?)
> >>
> >>
>
> > Zookeeper version is 3.4.5. This is part of a large software package, so
> > upgrading the solr library would be painful, but possible.
> >
>
> 3.4.5 is fine, I just wasn't sure what version solr was using, if it
> was old it might be related but I don't think that's the case here.
>
> >> "do not get autopurged" - could you provide more insight. 1) are you
> >> sure the configuration options are being applied? 2) what does the
> >> datadir look like? (snaps/logs, perhaps do an "ls -la" and provide in
> >> this thread) 3) anything autopurge related in the zk server logs?
> >>
>
> > Sure, the datadir contains only logs and snapshots. I would expect the
> > autoPurge function to delete logs and snapshots older than the number I
> > have set in zoo.cfg, but nothing gets deleted, and they just pile up,
> > eating up my hard drive.
> > 1) The zookeeper instance is picking up my settings for clientPort, so I
> > assume I have the purge settings in the correct place as well. The
> > relevant section of zoo.cfg:
> > autopurge.snapRetainCount=3
> > autopurge.purgeInterval=1
> >
> > 2) ls -lah
> > -rw-r--r-- 1 ryanfox staff 64M Oct 30 13:09 log.1
> > -rw-r--r-- 1 ryanfox staff 64M Nov 4 09:36 log.48
> > -rw-r--r-- 1 ryanfox staff 64M Nov 4 09:37 log.7b
> > -rw-r--r-- 1 ryanfox staff 296B Oct 30 12:53 snapshot.0
> > -rw-r--r-- 1 ryanfox staff 60K Nov 4 09:34 snapshot.47
> > -rw-r--r-- 1 ryanfox staff 60K Nov 4 09:36 snapshot.7a
> >
>
> You have our snapRetainCount set to 3. That's why three snaps (and
> their related logs) are saved.
>
> Try setting it to 2.
>
> > A new log and snapshot gets created every time I stop and start solr. As
> > an aside, every log file appears to be 64MB, no matter how much/little is
> > in my index. Is that correct? Seems like a lot for an empty solr
> instance.
> >
>
> We pre-allocate the log files for performance. See "preAllocSize" here:
> http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html
>
> If you want smaller files try setting that to something like 10k or
> somesuch.
>
> > 3) I haven't seen any smoking gun in the logs, but haven't specifically
> > looked, either. I'll let it go until it should have triggered the
> > auto-purge and report back.
>
> See if the above suggestions/info help.
>
> Obviously don't carry over these changes to production. ;-)
>
> Regards,
>
> Patrick
>
Re: auto-purge not purging
Posted by Patrick Hunt <ph...@apache.org>.
On Mon, Nov 4, 2013 at 7:46 AM, Ryan Fox <ry...@mediabeacon.com> wrote:
>>
>> What version of ZooKeeper are you running? (solr 4.1.0 is pretty old,
>> 4.5.1 is out now, perhaps try upgrading?)
>>
>>
> Zookeeper version is 3.4.5. This is part of a large software package, so
> upgrading the solr library would be painful, but possible.
>
3.4.5 is fine, I just wasn't sure what version solr was using, if it
was old it might be related but I don't think that's the case here.
>> "do not get autopurged" - could you provide more insight. 1) are you
>> sure the configuration options are being applied? 2) what does the
>> datadir look like? (snaps/logs, perhaps do an "ls -la" and provide in
>> this thread) 3) anything autopurge related in the zk server logs?
>>
> Sure, the datadir contains only logs and snapshots. I would expect the
> autoPurge function to delete logs and snapshots older than the number I
> have set in zoo.cfg, but nothing gets deleted, and they just pile up,
> eating up my hard drive.
> 1) The zookeeper instance is picking up my settings for clientPort, so I
> assume I have the purge settings in the correct place as well. The
> relevant section of zoo.cfg:
> autopurge.snapRetainCount=3
> autopurge.purgeInterval=1
>
> 2) ls -lah
> -rw-r--r-- 1 ryanfox staff 64M Oct 30 13:09 log.1
> -rw-r--r-- 1 ryanfox staff 64M Nov 4 09:36 log.48
> -rw-r--r-- 1 ryanfox staff 64M Nov 4 09:37 log.7b
> -rw-r--r-- 1 ryanfox staff 296B Oct 30 12:53 snapshot.0
> -rw-r--r-- 1 ryanfox staff 60K Nov 4 09:34 snapshot.47
> -rw-r--r-- 1 ryanfox staff 60K Nov 4 09:36 snapshot.7a
>
You have our snapRetainCount set to 3. That's why three snaps (and
their related logs) are saved.
Try setting it to 2.
> A new log and snapshot gets created every time I stop and start solr. As
> an aside, every log file appears to be 64MB, no matter how much/little is
> in my index. Is that correct? Seems like a lot for an empty solr instance.
>
We pre-allocate the log files for performance. See "preAllocSize" here:
http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html
If you want smaller files try setting that to something like 10k or somesuch.
> 3) I haven't seen any smoking gun in the logs, but haven't specifically
> looked, either. I'll let it go until it should have triggered the
> auto-purge and report back.
See if the above suggestions/info help.
Obviously don't carry over these changes to production. ;-)
Regards,
Patrick
Re: auto-purge not purging
Posted by Ryan Fox <ry...@mediabeacon.com>.
>
> What version of ZooKeeper are you running? (solr 4.1.0 is pretty old,
> 4.5.1 is out now, perhaps try upgrading?)
>
> Zookeeper version is 3.4.5. This is part of a large software package, so
upgrading the solr library would be painful, but possible.
"do not get autopurged" - could you provide more insight. 1) are you
> sure the configuration options are being applied? 2) what does the
> datadir look like? (snaps/logs, perhaps do an "ls -la" and provide in
> this thread) 3) anything autopurge related in the zk server logs?
>
> Sure, the datadir contains only logs and snapshots. I would expect the
autoPurge function to delete logs and snapshots older than the number I
have set in zoo.cfg, but nothing gets deleted, and they just pile up,
eating up my hard drive.
1) The zookeeper instance is picking up my settings for clientPort, so I
assume I have the purge settings in the correct place as well. The
relevant section of zoo.cfg:
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
2) ls -lah
-rw-r--r-- 1 ryanfox staff 64M Oct 30 13:09 log.1
-rw-r--r-- 1 ryanfox staff 64M Nov 4 09:36 log.48
-rw-r--r-- 1 ryanfox staff 64M Nov 4 09:37 log.7b
-rw-r--r-- 1 ryanfox staff 296B Oct 30 12:53 snapshot.0
-rw-r--r-- 1 ryanfox staff 60K Nov 4 09:34 snapshot.47
-rw-r--r-- 1 ryanfox staff 60K Nov 4 09:36 snapshot.7a
A new log and snapshot gets created every time I stop and start solr. As
an aside, every log file appears to be 64MB, no matter how much/little is
in my index. Is that correct? Seems like a lot for an empty solr instance.
3) I haven't seen any smoking gun in the logs, but haven't specifically
looked, either. I'll let it go until it should have triggered the
auto-purge and report back.
Thanks for your help in this.
Ryan
Re: auto-purge not purging
Posted by Patrick Hunt <ph...@apache.org>.
On Wed, Oct 30, 2013 at 8:05 AM, Ryan Fox <ry...@mediabeacon.com> wrote:
> Hello,
>
> I am running solr 4.1.0 with one shard and two replicas. One of my replica
What version of ZooKeeper are you running? (solr 4.1.0 is pretty old,
4.5.1 is out now, perhaps try upgrading?)
> nodes is running an embedded zookeeper instance. Per the administrator's
> guide, I have the properties `autopurge.snapRetainCount` and
> `autopurge.purgeInterval` set in zoo.cfg. Despite this, my transaction
> logs/snapshots do not get auto-purged, even after the interval has expired
> several times over.
"do not get autopurged" - could you provide more insight. 1) are you
sure the configuration options are being applied? 2) what does the
datadir look like? (snaps/logs, perhaps do an "ls -la" and provide in
this thread) 3) anything autopurge related in the zk server logs?
>
> Is this related to running zookeeper as an embedded process?
Unlikely.
Patrick