In Process Zookeeper Log Clean Up

["Mitchell Rathbun (BLOOMBERG/ 731 LEX)" <mr...@bloomberg.net>]

When running Storm using LocalCluster, it starts an in process zookeeper server. Normally, when running a zookeeper server, it is up to the administrator to remove old logs so the directory specified by datalogdir doesn't fill up. The zookeeper config settings autopurge.snapRetainCount and autopurge.purgeInterval allow this to be done automatically. I am wondering if Storm automatically handles the clean up of these logs or provides a config file for the above settings to be changed from defaults. From looking through the code, it seems that it doesn't. I am guessing this is because local mode is for testing and isn't really meant to support long running topologies. Does running Storm through LocalCluster provide a way to prevent Zookeeper server logs from overflowing? And in general, is LocalCluster meant to be used for long running topologies in production, or just testing?

Re: In Process Zookeeper Log Clean Up

[Manish Sharma <ma...@gmail.com>]

I am curious about your use case, Why are you running LocalCluster in
production?
You are missing out on distributed, high availability features when running
in cluster mode.


On Fri, Dec 8, 2017 at 2:37 PM, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <
mrathbun1@bloomberg.net> wrote:

> When running Storm using LocalCluster, it starts an in process zookeeper
> server. Normally, when running a zookeeper server, it is up to the
> administrator to remove old logs so the directory specified by datalogdir
> doesn't fill up. The zookeeper config settings autopurge.snapRetainCount
> and autopurge.purgeInterval allow this to be done automatically. I am
> wondering if Storm automatically handles the clean up of these logs or
> provides a config file for the above settings to be changed from defaults.
> From looking through the code, it seems that it doesn't. I am guessing this
> is because local mode is for testing and isn't really meant to support long
> running topologies. Does running Storm through LocalCluster provide a way
> to prevent Zookeeper server logs from overflowing? And in general, is
> LocalCluster meant to be used for long running topologies in production, or
> just testing?
>