You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Joe Stein <cr...@gmail.com> on 2011/11/04 16:46:24 UTC
HA - Kafka Server Setup?
Hey folks, I have been digging on and off the last couple of days to
figure/find out how to setup kafka server (the bus) so that I can have > 1
server with redundancies for my producers and consumers and how it all
works. I could have easily missed it with learning everything else about
the system. I guess in general this is something that I still have not
grocked and figured before I started to dig through the code for the first
time reach out to you folks to see if there was any information already put
together or tips/info I could find out or even just get pointed into the
code specifically where messages are replicated and stored. I also had a
bunch of questions in this regard to configurations for hardware best
practices based on usage but figure once I get it up and running I will
just run load tests and come up with my own needs for RAM, CPU and DISK
(and JVM configurations as such) based on # message and data for each
message.
Everything is very awesome so far and I think this is going to work
extremely well for a lot of what we do in production with a lot less copies
and no longer sequentially. I am also hoping to hook up an open source
consumer for Cassandra as we will be using Kafka day one for aggregating
metrics to Cassandra (which we have already started open sourcing which is
all Scala based).
Many thanx!
/*
Joe Stein
http://www.linkedin.com/in/charmalloc
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
*/
Re: HA - Kafka Server Setup?
Posted by Jun Rao <ju...@gmail.com>.
Hi, Joe,
Looking forward to your contribution. We plan to start working on
replication in a couple weeks. I will try to update the jira with some more
detailed info.
Thanks,
Jun
On Fri, Nov 4, 2011 at 11:53 AM, Joe Stein <cr...@gmail.com> wrote:
> Thanks for quick responses.
>
> Jun: KAFKA-50 looks pretty interesting. I am going to go through it in
> more detail again tonight. This feature not being in yet is not going to
> block me from getting going but will for wide spread use I think. Is there
> any opportunity to help contribute to this? Maybe start with something
> else (smaller) might be better to get my feet wet? I am going to start to
> cozy up to the code this weekend.... right now "Test Starting:
> testProduceAndMultiFetch(kafka.javaapi.integration.PrimitiveApiTest)" keeps
> hanging but maybe it resource issue on my machine (will try it on another
> machine and if still and issue will send to dev)
>
> Neha: I think as you said "just copy over the topic directories and start
> the Kafka cluster" for now will be a sufficient approach if/when a server
> dies since it sounds like another broker (Y) would get the requests when
> broker (X) dies (this is an assumption and easy enough for me to test).
> For the "at least" delivery guarantee I am assuming you are asking what we
> would do if we get another event that maybe we already got because of a
> failure? We actually deal with this type of thing a lot already (mostly
> from our mobile devices). It depends on the data that is flowing (in this
> paradigm I guess it would be the topic) and if N errors occurred within the
> last T seconds each processor has logic what todo (even some a
> system.exit())... with Kafka I would hope we would know when this happens
> meaning we know an error has occurred and we might for this one row get it
> again (in which case we could keep it somewhere resident to validate for Y
> time period).
>
> /*
> Joe Stein
> http://www.linkedin.com/in/charmalloc
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> */
>
> On Fri, Nov 4, 2011 at 12:41 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > >> i mean, the log segments
> > will always be in a consistent state on disk, right?
> >
> > Yes. You can just copy over the topic directories and start the Kafka
> > cluster
> >
> > Thanks,
> > Neha
> >
> > On Fri, Nov 4, 2011 at 9:35 AM, Tim Lossen <ti...@lossen.de> wrote:
> > > ok, but apart from the possibility of data loss, rsync
> > > in principle should work fine? i mean, the log segments
> > > will always be in a consistent state on disk, right?
> > >
> > >
> > > On 2011-11-04, at 17:18 , Neha Narkhede wrote:
> > >
> > >>>> for redundancy, we were planning to simply rsync the kafka
> > >> message logs to a second machine periodically. or do you
> > >> see any obvious problems with that?
> > >>
> > >> Rsync approach has several problems and could be a lossy solution. We
> > >> moved away from that by replacing a legacy system with Kafka.
> > >> We recommend you setup your redundant cluster using the mirroring
> > >> approach, which is much more reliable and real-time than rsync.
> > >>
> > >> I think 0.6 has a stripped down version of mirroring, where you cannot
> > >> control the mirroring for specific topics.
> > >>
> > >> Thanks,
> > >> Neha
> > >>
> > >> On Fri, Nov 4, 2011 at 9:12 AM, Tim Lossen <ti...@lossen.de> wrote:
> > >>>
> > >>> interesting. is this already available in 0.6?
> > >>>
> > >>> for redundancy, we were planning to simply rsync the kafka
> > >>> message logs to a second machine periodically. or do you
> > >>> see any obvious problems with that?
> > >>>
> > >>> cheers
> > >>> tim
> > >>>
> > >>> On 2011-11-04, at 17:07 , Neha Narkhede wrote:
> > >>>> We have a mirroring feature where you can setup 2 clusters, one to
> be
> > a
> > >>>> mirror of the other. At LinkedIn, we have a production Kafka
> cluster,
> > and
> > >>>> an analytics Kafka cluster that mirrors the production one in real
> > time. We
> > >>>> still haven't updated the documentation to describe this in detail.
> > >>>
> > >>> --
> > >>> http://tim.lossen.de
> > >>>
> > >>>
> > >>>
> > >
> > > --
> > > http://tim.lossen.de
> > >
> > >
> > >
> > >
> >
>
Re: HA - Kafka Server Setup?
Posted by Jay Kreps <ja...@gmail.com>.
We would love to have help! There is a tree of JIRAs now under KAFKA-50
that give some break down of the work but they are very very sparse and
don't capture the dependencies between items. Some work has started on
(KAFKA-48), but for the last month or so we linkedin folks have been
focusing on compression and the 0.7 release as well as our internal rollout
of 0.7 across our clusters. Jun could chime in with more details, our
roadmap looks a little like this:
- This week we are working on 0.7 open source release
- The next two weeks (I think) we had planned work on a few operational
issues and bugs we are hitting as well as go to ApacheCon
- Then we will start the replication work in earnest full time as our
primary focus
We would *love* to get some help on this--it is a fairly big feature and
could definitely benefit from more brains and hands as well as just people
thinking through different trade-offs against their own use cases.
So maybe a good plan in the neartime (say next week) is try to expand out
the work in more detail. I think it would be useful to get the JIRAs to
have a lot more details and work out some of the dependencies between
projects. Once the dependencies are worked out I think any of those JIRAs
could be picked up by any interested person and we can start to do the
design and implementation discussions on that aspect.
Jun, does that sound reasonable? Did I correctly summarize your plan?
-Jay
On Fri, Nov 4, 2011 at 11:53 AM, Joe Stein <cr...@gmail.com> wrote:
> Thanks for quick responses.
>
> Jun: KAFKA-50 looks pretty interesting. I am going to go through it in
> more detail again tonight. This feature not being in yet is not going to
> block me from getting going but will for wide spread use I think. Is there
> any opportunity to help contribute to this? Maybe start with something
> else (smaller) might be better to get my feet wet? I am going to start to
> cozy up to the code this weekend.... right now "Test Starting:
> testProduceAndMultiFetch(kafka.javaapi.integration.PrimitiveApiTest)" keeps
> hanging but maybe it resource issue on my machine (will try it on another
> machine and if still and issue will send to dev)
>
> Neha: I think as you said "just copy over the topic directories and start
> the Kafka cluster" for now will be a sufficient approach if/when a server
> dies since it sounds like another broker (Y) would get the requests when
> broker (X) dies (this is an assumption and easy enough for me to test).
> For the "at least" delivery guarantee I am assuming you are asking what we
> would do if we get another event that maybe we already got because of a
> failure? We actually deal with this type of thing a lot already (mostly
> from our mobile devices). It depends on the data that is flowing (in this
> paradigm I guess it would be the topic) and if N errors occurred within the
> last T seconds each processor has logic what todo (even some a
> system.exit())... with Kafka I would hope we would know when this happens
> meaning we know an error has occurred and we might for this one row get it
> again (in which case we could keep it somewhere resident to validate for Y
> time period).
>
> /*
> Joe Stein
> http://www.linkedin.com/in/charmalloc
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> */
>
> On Fri, Nov 4, 2011 at 12:41 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > >> i mean, the log segments
> > will always be in a consistent state on disk, right?
> >
> > Yes. You can just copy over the topic directories and start the Kafka
> > cluster
> >
> > Thanks,
> > Neha
> >
> > On Fri, Nov 4, 2011 at 9:35 AM, Tim Lossen <ti...@lossen.de> wrote:
> > > ok, but apart from the possibility of data loss, rsync
> > > in principle should work fine? i mean, the log segments
> > > will always be in a consistent state on disk, right?
> > >
> > >
> > > On 2011-11-04, at 17:18 , Neha Narkhede wrote:
> > >
> > >>>> for redundancy, we were planning to simply rsync the kafka
> > >> message logs to a second machine periodically. or do you
> > >> see any obvious problems with that?
> > >>
> > >> Rsync approach has several problems and could be a lossy solution. We
> > >> moved away from that by replacing a legacy system with Kafka.
> > >> We recommend you setup your redundant cluster using the mirroring
> > >> approach, which is much more reliable and real-time than rsync.
> > >>
> > >> I think 0.6 has a stripped down version of mirroring, where you cannot
> > >> control the mirroring for specific topics.
> > >>
> > >> Thanks,
> > >> Neha
> > >>
> > >> On Fri, Nov 4, 2011 at 9:12 AM, Tim Lossen <ti...@lossen.de> wrote:
> > >>>
> > >>> interesting. is this already available in 0.6?
> > >>>
> > >>> for redundancy, we were planning to simply rsync the kafka
> > >>> message logs to a second machine periodically. or do you
> > >>> see any obvious problems with that?
> > >>>
> > >>> cheers
> > >>> tim
> > >>>
> > >>> On 2011-11-04, at 17:07 , Neha Narkhede wrote:
> > >>>> We have a mirroring feature where you can setup 2 clusters, one to
> be
> > a
> > >>>> mirror of the other. At LinkedIn, we have a production Kafka
> cluster,
> > and
> > >>>> an analytics Kafka cluster that mirrors the production one in real
> > time. We
> > >>>> still haven't updated the documentation to describe this in detail.
> > >>>
> > >>> --
> > >>> http://tim.lossen.de
> > >>>
> > >>>
> > >>>
> > >
> > > --
> > > http://tim.lossen.de
> > >
> > >
> > >
> > >
> >
>
Re: HA - Kafka Server Setup?
Posted by Joe Stein <cr...@gmail.com>.
Thanks for quick responses.
Jun: KAFKA-50 looks pretty interesting. I am going to go through it in
more detail again tonight. This feature not being in yet is not going to
block me from getting going but will for wide spread use I think. Is there
any opportunity to help contribute to this? Maybe start with something
else (smaller) might be better to get my feet wet? I am going to start to
cozy up to the code this weekend.... right now "Test Starting:
testProduceAndMultiFetch(kafka.javaapi.integration.PrimitiveApiTest)" keeps
hanging but maybe it resource issue on my machine (will try it on another
machine and if still and issue will send to dev)
Neha: I think as you said "just copy over the topic directories and start
the Kafka cluster" for now will be a sufficient approach if/when a server
dies since it sounds like another broker (Y) would get the requests when
broker (X) dies (this is an assumption and easy enough for me to test).
For the "at least" delivery guarantee I am assuming you are asking what we
would do if we get another event that maybe we already got because of a
failure? We actually deal with this type of thing a lot already (mostly
from our mobile devices). It depends on the data that is flowing (in this
paradigm I guess it would be the topic) and if N errors occurred within the
last T seconds each processor has logic what todo (even some a
system.exit())... with Kafka I would hope we would know when this happens
meaning we know an error has occurred and we might for this one row get it
again (in which case we could keep it somewhere resident to validate for Y
time period).
/*
Joe Stein
http://www.linkedin.com/in/charmalloc
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
*/
On Fri, Nov 4, 2011 at 12:41 PM, Neha Narkhede <ne...@gmail.com>wrote:
> >> i mean, the log segments
> will always be in a consistent state on disk, right?
>
> Yes. You can just copy over the topic directories and start the Kafka
> cluster
>
> Thanks,
> Neha
>
> On Fri, Nov 4, 2011 at 9:35 AM, Tim Lossen <ti...@lossen.de> wrote:
> > ok, but apart from the possibility of data loss, rsync
> > in principle should work fine? i mean, the log segments
> > will always be in a consistent state on disk, right?
> >
> >
> > On 2011-11-04, at 17:18 , Neha Narkhede wrote:
> >
> >>>> for redundancy, we were planning to simply rsync the kafka
> >> message logs to a second machine periodically. or do you
> >> see any obvious problems with that?
> >>
> >> Rsync approach has several problems and could be a lossy solution. We
> >> moved away from that by replacing a legacy system with Kafka.
> >> We recommend you setup your redundant cluster using the mirroring
> >> approach, which is much more reliable and real-time than rsync.
> >>
> >> I think 0.6 has a stripped down version of mirroring, where you cannot
> >> control the mirroring for specific topics.
> >>
> >> Thanks,
> >> Neha
> >>
> >> On Fri, Nov 4, 2011 at 9:12 AM, Tim Lossen <ti...@lossen.de> wrote:
> >>>
> >>> interesting. is this already available in 0.6?
> >>>
> >>> for redundancy, we were planning to simply rsync the kafka
> >>> message logs to a second machine periodically. or do you
> >>> see any obvious problems with that?
> >>>
> >>> cheers
> >>> tim
> >>>
> >>> On 2011-11-04, at 17:07 , Neha Narkhede wrote:
> >>>> We have a mirroring feature where you can setup 2 clusters, one to be
> a
> >>>> mirror of the other. At LinkedIn, we have a production Kafka cluster,
> and
> >>>> an analytics Kafka cluster that mirrors the production one in real
> time. We
> >>>> still haven't updated the documentation to describe this in detail.
> >>>
> >>> --
> >>> http://tim.lossen.de
> >>>
> >>>
> >>>
> >
> > --
> > http://tim.lossen.de
> >
> >
> >
> >
>
Re: HA - Kafka Server Setup?
Posted by Neha Narkhede <ne...@gmail.com>.
>> i mean, the log segments
will always be in a consistent state on disk, right?
Yes. You can just copy over the topic directories and start the Kafka cluster
Thanks,
Neha
On Fri, Nov 4, 2011 at 9:35 AM, Tim Lossen <ti...@lossen.de> wrote:
> ok, but apart from the possibility of data loss, rsync
> in principle should work fine? i mean, the log segments
> will always be in a consistent state on disk, right?
>
>
> On 2011-11-04, at 17:18 , Neha Narkhede wrote:
>
>>>> for redundancy, we were planning to simply rsync the kafka
>> message logs to a second machine periodically. or do you
>> see any obvious problems with that?
>>
>> Rsync approach has several problems and could be a lossy solution. We
>> moved away from that by replacing a legacy system with Kafka.
>> We recommend you setup your redundant cluster using the mirroring
>> approach, which is much more reliable and real-time than rsync.
>>
>> I think 0.6 has a stripped down version of mirroring, where you cannot
>> control the mirroring for specific topics.
>>
>> Thanks,
>> Neha
>>
>> On Fri, Nov 4, 2011 at 9:12 AM, Tim Lossen <ti...@lossen.de> wrote:
>>>
>>> interesting. is this already available in 0.6?
>>>
>>> for redundancy, we were planning to simply rsync the kafka
>>> message logs to a second machine periodically. or do you
>>> see any obvious problems with that?
>>>
>>> cheers
>>> tim
>>>
>>> On 2011-11-04, at 17:07 , Neha Narkhede wrote:
>>>> We have a mirroring feature where you can setup 2 clusters, one to be a
>>>> mirror of the other. At LinkedIn, we have a production Kafka cluster, and
>>>> an analytics Kafka cluster that mirrors the production one in real time. We
>>>> still haven't updated the documentation to describe this in detail.
>>>
>>> --
>>> http://tim.lossen.de
>>>
>>>
>>>
>
> --
> http://tim.lossen.de
>
>
>
>
Re: HA - Kafka Server Setup?
Posted by Tim Lossen <ti...@lossen.de>.
ok, but apart from the possibility of data loss, rsync
in principle should work fine? i mean, the log segments
will always be in a consistent state on disk, right?
On 2011-11-04, at 17:18 , Neha Narkhede wrote:
>>> for redundancy, we were planning to simply rsync the kafka
> message logs to a second machine periodically. or do you
> see any obvious problems with that?
>
> Rsync approach has several problems and could be a lossy solution. We
> moved away from that by replacing a legacy system with Kafka.
> We recommend you setup your redundant cluster using the mirroring
> approach, which is much more reliable and real-time than rsync.
>
> I think 0.6 has a stripped down version of mirroring, where you cannot
> control the mirroring for specific topics.
>
> Thanks,
> Neha
>
> On Fri, Nov 4, 2011 at 9:12 AM, Tim Lossen <ti...@lossen.de> wrote:
>>
>> interesting. is this already available in 0.6?
>>
>> for redundancy, we were planning to simply rsync the kafka
>> message logs to a second machine periodically. or do you
>> see any obvious problems with that?
>>
>> cheers
>> tim
>>
>> On 2011-11-04, at 17:07 , Neha Narkhede wrote:
>>> We have a mirroring feature where you can setup 2 clusters, one to be a
>>> mirror of the other. At LinkedIn, we have a production Kafka cluster, and
>>> an analytics Kafka cluster that mirrors the production one in real time. We
>>> still haven't updated the documentation to describe this in detail.
>>
>> --
>> http://tim.lossen.de
>>
>>
>>
--
http://tim.lossen.de
Re: HA - Kafka Server Setup?
Posted by Neha Narkhede <ne...@gmail.com>.
>>for redundancy, we were planning to simply rsync the kafka
message logs to a second machine periodically. or do you
see any obvious problems with that?
Rsync approach has several problems and could be a lossy solution. We
moved away from that by replacing a legacy system with Kafka.
We recommend you setup your redundant cluster using the mirroring
approach, which is much more reliable and real-time than rsync.
I think 0.6 has a stripped down version of mirroring, where you cannot
control the mirroring for specific topics.
Thanks,
Neha
On Fri, Nov 4, 2011 at 9:12 AM, Tim Lossen <ti...@lossen.de> wrote:
>
> interesting. is this already available in 0.6?
>
> for redundancy, we were planning to simply rsync the kafka
> message logs to a second machine periodically. or do you
> see any obvious problems with that?
>
> cheers
> tim
>
> On 2011-11-04, at 17:07 , Neha Narkhede wrote:
> > We have a mirroring feature where you can setup 2 clusters, one to be a
> > mirror of the other. At LinkedIn, we have a production Kafka cluster, and
> > an analytics Kafka cluster that mirrors the production one in real time. We
> > still haven't updated the documentation to describe this in detail.
>
> --
> http://tim.lossen.de
>
>
>
Re: HA - Kafka Server Setup?
Posted by Tim Lossen <ti...@lossen.de>.
interesting. is this already available in 0.6?
for redundancy, we were planning to simply rsync the kafka
message logs to a second machine periodically. or do you
see any obvious problems with that?
cheers
tim
On 2011-11-04, at 17:07 , Neha Narkhede wrote:
> We have a mirroring feature where you can setup 2 clusters, one to be a
> mirror of the other. At LinkedIn, we have a production Kafka cluster, and
> an analytics Kafka cluster that mirrors the production one in real time. We
> still haven't updated the documentation to describe this in detail.
--
http://tim.lossen.de
Re: HA - Kafka Server Setup?
Posted by Neha Narkhede <ne...@gmail.com>.
>> how to setup kafka server (the bus) so that I can have > 1
server with redundancies for my producers and consumers and how it all
works.
We have a mirroring feature where you can setup 2 clusters, one to be a
mirror of the other. At LinkedIn, we have a production Kafka cluster, and
an analytics Kafka cluster that mirrors the production one in real time. We
still haven't updated the documentation to describe this in detail.
Until replication is out, does the above mirroring approach work for you ?
>> I am also hoping to hook up an open source
consumer for Cassandra as we will be using Kafka day one for aggregating
metrics to Cassandra (which we have already started open sourcing which is
all Scala based).
Exciting ! Let us know how that goes for you. A question I have - how do
you think you will handle the "at least" delivery guarantees of Kafka here ?
Thanks,
Neha
On Fri, Nov 4, 2011 at 8:56 AM, Jun Rao <ju...@gmail.com> wrote:
> Joe,
>
> Setting up multiple brokers on different machines is pretty easy. You just
> need to give each broker a unique broker id and give them the same ZK
> connect string. By default, producers will spread the data randomly among
> all brokers. There is currently no redundancy across brokers. So if a
> broker is permanently gone, all unconsumed data on it is lost. We will be
> working on replication (KAFKA-50) in the next release.
>
> Thanks,
>
> Jun
>
> On Fri, Nov 4, 2011 at 8:46 AM, Joe Stein <cr...@gmail.com> wrote:
>
> > Hey folks, I have been digging on and off the last couple of days to
> > figure/find out how to setup kafka server (the bus) so that I can have >
> 1
> > server with redundancies for my producers and consumers and how it all
> > works. I could have easily missed it with learning everything else about
> > the system. I guess in general this is something that I still have not
> > grocked and figured before I started to dig through the code for the
> first
> > time reach out to you folks to see if there was any information already
> put
> > together or tips/info I could find out or even just get pointed into the
> > code specifically where messages are replicated and stored. I also had a
> > bunch of questions in this regard to configurations for hardware best
> > practices based on usage but figure once I get it up and running I will
> > just run load tests and come up with my own needs for RAM, CPU and DISK
> > (and JVM configurations as such) based on # message and data for each
> > message.
> >
> > Everything is very awesome so far and I think this is going to work
> > extremely well for a lot of what we do in production with a lot less
> copies
> > and no longer sequentially. I am also hoping to hook up an open source
> > consumer for Cassandra as we will be using Kafka day one for aggregating
> > metrics to Cassandra (which we have already started open sourcing which
> is
> > all Scala based).
> >
> > Many thanx!
> >
> > /*
> > Joe Stein
> > http://www.linkedin.com/in/charmalloc
> > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > */
> >
>
Re: HA - Kafka Server Setup?
Posted by Jun Rao <ju...@gmail.com>.
Joe,
Setting up multiple brokers on different machines is pretty easy. You just
need to give each broker a unique broker id and give them the same ZK
connect string. By default, producers will spread the data randomly among
all brokers. There is currently no redundancy across brokers. So if a
broker is permanently gone, all unconsumed data on it is lost. We will be
working on replication (KAFKA-50) in the next release.
Thanks,
Jun
On Fri, Nov 4, 2011 at 8:46 AM, Joe Stein <cr...@gmail.com> wrote:
> Hey folks, I have been digging on and off the last couple of days to
> figure/find out how to setup kafka server (the bus) so that I can have > 1
> server with redundancies for my producers and consumers and how it all
> works. I could have easily missed it with learning everything else about
> the system. I guess in general this is something that I still have not
> grocked and figured before I started to dig through the code for the first
> time reach out to you folks to see if there was any information already put
> together or tips/info I could find out or even just get pointed into the
> code specifically where messages are replicated and stored. I also had a
> bunch of questions in this regard to configurations for hardware best
> practices based on usage but figure once I get it up and running I will
> just run load tests and come up with my own needs for RAM, CPU and DISK
> (and JVM configurations as such) based on # message and data for each
> message.
>
> Everything is very awesome so far and I think this is going to work
> extremely well for a lot of what we do in production with a lot less copies
> and no longer sequentially. I am also hoping to hook up an open source
> consumer for Cassandra as we will be using Kafka day one for aggregating
> metrics to Cassandra (which we have already started open sourcing which is
> all Scala based).
>
> Many thanx!
>
> /*
> Joe Stein
> http://www.linkedin.com/in/charmalloc
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> */
>