You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Zhou Yu <Zh...@thehutgroup.com> on 2014/08/12 10:23:49 UTC
Zookeeper client transactional logs
Hi all,
I am trying to set up a apache storm cluster with storm kafka using zookeeper. Everything is running fine, expect I see
huge unreadable files being created while the cluster is running. They are located in /tmp folder, along with the zookeeper
dir, they are named similarly to
f2291425-a2d5-490b-91fc-3f886801d39f
and there a few of them. The folder with the biggest files has a subdirectory called version-2, and the files are named similarly
to
log.1fda8
Do you guys know what logs they are and how I can deal with them if I don't want them to bloat my disk? (I tried removing them
periodically but then at some point the storm cluster crashed due to not finding the file).
Thanks a lot!
Zhou Yu
IT Intern
The Hut Group<http://www.thehutgroup.com/>
Tel:
Email: Zhou.Yu@thehutgroup.com<ma...@thehutgroup.com>
For the purposes of this email, the "company" means The Hut Group Limited, a company registered in England and Wales (company number 6539496) whose registered office is at Meridian House, Gadbrook Park, Rudheath, Northwich, Cheshire, CW9 7RA and/or any of its respective subsidiaries.
Confidentiality Notice
This e-mail is confidential and intended for the use of the named recipient only. If you are not the intended recipient please notify us by telephone immediately on +44(0)1606 338197 or return it to us by e-mail. Please then delete it from your system and note that any use, dissemination, forwarding, printing or copying is strictly prohibited. Any views or opinions are solely those of the author and do not necessarily represent those of the company.
Encryptions and Viruses
Please note that this e-mail and any attachments have not been encrypted. They may therefore be liable to be compromised. Please also note that it is your responsibility to scan this e-mail and any attachments for viruses. We do not, to the extent permitted by law, accept any liability (whether in contract, negligence or otherwise) for any virus infection and/or external compromise of security and/or confidentiality in relation to transmissions sent by e-mail.
Monitoring
Activity and use of the company's systems is monitored to secure its effective use and operation and for other lawful business purposes. Communications using these systems will also be monitored and may be recorded to secure effective use and operation and for other lawful business purposes.
hgvyjuv
RE: Zookeeper client transactional logs
Posted by Zhou Yu <Zh...@thehutgroup.com>.
Hi Michi,
Thanks for your reply! Very appreciated! I had a look at the page you sent me yesterday, but was a bit confused
why the logs are on the client side. As for my application, I don't think that there is a zookeeper server running.
If that's the case, why does the client need to keep the transaction log files?
________________________________________
From: mutsuzaki@gmail.com [mutsuzaki@gmail.com] on behalf of Michi Mutsuzaki [michi@cs.stanford.edu]
Sent: 12 August 2014 09:36
To: user@zookeeper.apache.org
Subject: Re: Zookeeper client transactional logs
Hi Zhou,
These are ZooKeeper transaction log files. ZooKeeper needs the latest
snapshot file and the latest transaction log file to function
properly. You can use
autopurge.snapRetainCount to automatically delete the ones you don't need.
http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#sc_maintenance
On Tue, Aug 12, 2014 at 1:23 AM, Zhou Yu <Zh...@thehutgroup.com> wrote:
> Hi all,
>
> I am trying to set up a apache storm cluster with storm kafka using zookeeper. Everything is running fine, expect I see
> huge unreadable files being created while the cluster is running. They are located in /tmp folder, along with the zookeeper
> dir, they are named similarly to
>
> f2291425-a2d5-490b-91fc-3f886801d39f
>
> and there a few of them. The folder with the biggest files has a subdirectory called version-2, and the files are named similarly
> to
>
> log.1fda8
>
> Do you guys know what logs they are and how I can deal with them if I don't want them to bloat my disk? (I tried removing them
> periodically but then at some point the storm cluster crashed due to not finding the file).
>
> Thanks a lot!
> Zhou Yu
> IT Intern
> The Hut Group<http://www.thehutgroup.com/>
>
> Tel:
> Email: Zhou.Yu@thehutgroup.com<ma...@thehutgroup.com>
>
> For the purposes of this email, the "company" means The Hut Group Limited, a company registered in England and Wales (company number 6539496) whose registered office is at Meridian House, Gadbrook Park, Rudheath, Northwich, Cheshire, CW9 7RA and/or any of its respective subsidiaries.
>
> Confidentiality Notice
> This e-mail is confidential and intended for the use of the named recipient only. If you are not the intended recipient please notify us by telephone immediately on +44(0)1606 338197 or return it to us by e-mail. Please then delete it from your system and note that any use, dissemination, forwarding, printing or copying is strictly prohibited. Any views or opinions are solely those of the author and do not necessarily represent those of the company.
>
> Encryptions and Viruses
> Please note that this e-mail and any attachments have not been encrypted. They may therefore be liable to be compromised. Please also note that it is your responsibility to scan this e-mail and any attachments for viruses. We do not, to the extent permitted by law, accept any liability (whether in contract, negligence or otherwise) for any virus infection and/or external compromise of security and/or confidentiality in relation to transmissions sent by e-mail.
>
> Monitoring
> Activity and use of the company's systems is monitored to secure its effective use and operation and for other lawful business purposes. Communications using these systems will also be monitored and may be recorded to secure effective use and operation and for other lawful business purposes.
>
> hgvyjuv
Re: Zookeeper client transactional logs
Posted by Michi Mutsuzaki <mi...@cs.stanford.edu>.
> 1. Can you point out where I can change the directory for the logs is specified, and how I can modify what it
> logs?
You can set dataDir to specify the log directory. You cannot modify
what ZooKeeper logs.
http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#sc_configuration
> 2. Is it possible to reduce the number of logs?
Have you tried using autopurge.snapRetainCount?
> 3. Why does the client need these logs anyway?
I'm pretty sure you are running ZooKeeper server there. Maybe you can
post your ZooKeeper configuration file.
RE: Zookeeper client transactional logs
Posted by Zhou Yu <Zh...@thehutgroup.com>.
Hi Michi,
Sorry to bother you again. I am not sure if you're familiar with storm kafka project, but I am having problems
with the transaction log files filling up my client side. A few questions:
1. Can you point out where I can change the directory for the logs is specified, and how I can modify what it
logs?
2. Is it possible to reduce the number of logs?
3. Why does the client need these logs anyway?
________________________________________
From: mutsuzaki@gmail.com [mutsuzaki@gmail.com] on behalf of Michi Mutsuzaki [michi@cs.stanford.edu]
Sent: 12 August 2014 09:36
To: user@zookeeper.apache.org
Subject: Re: Zookeeper client transactional logs
Hi Zhou,
These are ZooKeeper transaction log files. ZooKeeper needs the latest
snapshot file and the latest transaction log file to function
properly. You can use
autopurge.snapRetainCount to automatically delete the ones you don't need.
http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#sc_maintenance
On Tue, Aug 12, 2014 at 1:23 AM, Zhou Yu <Zh...@thehutgroup.com> wrote:
> Hi all,
>
> I am trying to set up a apache storm cluster with storm kafka using zookeeper. Everything is running fine, expect I see
> huge unreadable files being created while the cluster is running. They are located in /tmp folder, along with the zookeeper
> dir, they are named similarly to
>
> f2291425-a2d5-490b-91fc-3f886801d39f
>
> and there a few of them. The folder with the biggest files has a subdirectory called version-2, and the files are named similarly
> to
>
> log.1fda8
>
> Do you guys know what logs they are and how I can deal with them if I don't want them to bloat my disk? (I tried removing them
> periodically but then at some point the storm cluster crashed due to not finding the file).
>
> Thanks a lot!
> Zhou Yu
> IT Intern
> The Hut Group<http://www.thehutgroup.com/>
>
> Tel:
> Email: Zhou.Yu@thehutgroup.com<ma...@thehutgroup.com>
>
> For the purposes of this email, the "company" means The Hut Group Limited, a company registered in England and Wales (company number 6539496) whose registered office is at Meridian House, Gadbrook Park, Rudheath, Northwich, Cheshire, CW9 7RA and/or any of its respective subsidiaries.
>
> Confidentiality Notice
> This e-mail is confidential and intended for the use of the named recipient only. If you are not the intended recipient please notify us by telephone immediately on +44(0)1606 338197 or return it to us by e-mail. Please then delete it from your system and note that any use, dissemination, forwarding, printing or copying is strictly prohibited. Any views or opinions are solely those of the author and do not necessarily represent those of the company.
>
> Encryptions and Viruses
> Please note that this e-mail and any attachments have not been encrypted. They may therefore be liable to be compromised. Please also note that it is your responsibility to scan this e-mail and any attachments for viruses. We do not, to the extent permitted by law, accept any liability (whether in contract, negligence or otherwise) for any virus infection and/or external compromise of security and/or confidentiality in relation to transmissions sent by e-mail.
>
> Monitoring
> Activity and use of the company's systems is monitored to secure its effective use and operation and for other lawful business purposes. Communications using these systems will also be monitored and may be recorded to secure effective use and operation and for other lawful business purposes.
>
> hgvyjuv
Re: Zookeeper client transactional logs
Posted by Michi Mutsuzaki <mi...@cs.stanford.edu>.
Hi Zhou,
These are ZooKeeper transaction log files. ZooKeeper needs the latest
snapshot file and the latest transaction log file to function
properly. You can use
autopurge.snapRetainCount to automatically delete the ones you don't need.
http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#sc_maintenance
On Tue, Aug 12, 2014 at 1:23 AM, Zhou Yu <Zh...@thehutgroup.com> wrote:
> Hi all,
>
> I am trying to set up a apache storm cluster with storm kafka using zookeeper. Everything is running fine, expect I see
> huge unreadable files being created while the cluster is running. They are located in /tmp folder, along with the zookeeper
> dir, they are named similarly to
>
> f2291425-a2d5-490b-91fc-3f886801d39f
>
> and there a few of them. The folder with the biggest files has a subdirectory called version-2, and the files are named similarly
> to
>
> log.1fda8
>
> Do you guys know what logs they are and how I can deal with them if I don't want them to bloat my disk? (I tried removing them
> periodically but then at some point the storm cluster crashed due to not finding the file).
>
> Thanks a lot!
> Zhou Yu
> IT Intern
> The Hut Group<http://www.thehutgroup.com/>
>
> Tel:
> Email: Zhou.Yu@thehutgroup.com<ma...@thehutgroup.com>
>
> For the purposes of this email, the "company" means The Hut Group Limited, a company registered in England and Wales (company number 6539496) whose registered office is at Meridian House, Gadbrook Park, Rudheath, Northwich, Cheshire, CW9 7RA and/or any of its respective subsidiaries.
>
> Confidentiality Notice
> This e-mail is confidential and intended for the use of the named recipient only. If you are not the intended recipient please notify us by telephone immediately on +44(0)1606 338197 or return it to us by e-mail. Please then delete it from your system and note that any use, dissemination, forwarding, printing or copying is strictly prohibited. Any views or opinions are solely those of the author and do not necessarily represent those of the company.
>
> Encryptions and Viruses
> Please note that this e-mail and any attachments have not been encrypted. They may therefore be liable to be compromised. Please also note that it is your responsibility to scan this e-mail and any attachments for viruses. We do not, to the extent permitted by law, accept any liability (whether in contract, negligence or otherwise) for any virus infection and/or external compromise of security and/or confidentiality in relation to transmissions sent by e-mail.
>
> Monitoring
> Activity and use of the company's systems is monitored to secure its effective use and operation and for other lawful business purposes. Communications using these systems will also be monitored and may be recorded to secure effective use and operation and for other lawful business purposes.
>
> hgvyjuv