You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mike Marzo <pr...@gmail.com> on 2016/12/08 12:33:51 UTC
Kafka and zookeeper stores and mesos env
If i'm running a 5 node zk cluster and a 3 node kafka cluster in dcker on a
mesos/marathon environment where my zk and broker nodes are all leveraging
local disk on the hosts they are running on is there any value to the local
data being preserved across restarts?
In other words.... when a broker node fails and restarts on the same
machine does it leverage any of the actual data it has on disk from the
prior life or is it assumed to start from new being re-hydrated from the
other in sync nodes in the running cluster? Same question for
zookeeper...
I'm trying to asses the value of local data preservation across marathon re
scheduled jobs. Im thinking the data is blown away as much would be stale
but not sure since log retentions could be set quite high and having
history could make re-sync more efficient by effectively only closing the
gap for deltas while down. Anyone know
Re: Kafka and zookeeper stores and mesos env
Posted by Mike Marzo <pr...@gmail.com>.
understood, and i am looking at that bit but i would still like to know the
answer.
On Thu, Dec 8, 2016 at 8:22 AM, Asaf Mesika <as...@gmail.com> wrote:
> Off-question a bit - Using the Kafka Mesos framework should save you from
> handling those questions: https://github.com/mesos/kafka
>
>
> On Thu, Dec 8, 2016 at 2:33 PM Mike Marzo <pr...@gmail.com>
> wrote:
>
> If i'm running a 5 node zk cluster and a 3 node kafka cluster in dcker on a
> mesos/marathon environment where my zk and broker nodes are all leveraging
> local disk on the hosts they are running on is there any value to the local
> data being preserved across restarts?
>
> In other words.... when a broker node fails and restarts on the same
> machine does it leverage any of the actual data it has on disk from the
> prior life or is it assumed to start from new being re-hydrated from the
> other in sync nodes in the running cluster? Same question for
> zookeeper...
>
> I'm trying to asses the value of local data preservation across marathon re
> scheduled jobs. Im thinking the data is blown away as much would be stale
> but not sure since log retentions could be set quite high and having
> history could make re-sync more efficient by effectively only closing the
> gap for deltas while down. Anyone know
>
Re: Kafka and zookeeper stores and mesos env
Posted by Asaf Mesika <as...@gmail.com>.
Off-question a bit - Using the Kafka Mesos framework should save you from
handling those questions: https://github.com/mesos/kafka
On Thu, Dec 8, 2016 at 2:33 PM Mike Marzo <pr...@gmail.com>
wrote:
If i'm running a 5 node zk cluster and a 3 node kafka cluster in dcker on a
mesos/marathon environment where my zk and broker nodes are all leveraging
local disk on the hosts they are running on is there any value to the local
data being preserved across restarts?
In other words.... when a broker node fails and restarts on the same
machine does it leverage any of the actual data it has on disk from the
prior life or is it assumed to start from new being re-hydrated from the
other in sync nodes in the running cluster? Same question for
zookeeper...
I'm trying to asses the value of local data preservation across marathon re
scheduled jobs. Im thinking the data is blown away as much would be stale
but not sure since log retentions could be set quite high and having
history could make re-sync more efficient by effectively only closing the
gap for deltas while down. Anyone know