You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kyle Mathews <ma...@gmail.com> on 2016/02/15 01:07:19 UTC
Kafka advice for small startup
Hi I'm the technical co-founder of a startup in San Francisco and I've been
investigating using Kafka for some time.
I'm thinking about biting the bullet and integrating it soon into our
system. My main concern is how much operational overhead it'll add. I'm the
only engineer right now and am already stretched thin. Since Kafka isn't a
must have yet, it only makes sense to adopt it if it's pretty care free.
What are people's experiences in similar situations to mine? Is Kafka
something you can setup and forget like say redis or is something I'll be
babysitting a decent bit and spending a lot of time on upgrades etc.?
Also I'd love to take anyone in San Francisco with Kafka experience out to
lunch or coffee.
Thanks!
Kyle
Re: Kafka advice for small startup
Posted by Sharninder <sh...@gmail.com>.
Kafka is pretty nice and as long as you have basic monitoring in place,
doesn't take too much attention, but keep in mind that it still depends on
zookeeper and I've seen that being the bottleneck. in the past. I also
think as a single engineering person in your startup, if you don't need
kafka or any other component, don't get it. All said and done, it still is
one extra component to manage.
--
Sharninder
On Mon, Feb 15, 2016 at 6:07 AM, Todd Snyder <ts...@blackberry.com> wrote:
> So long as you put some basic monitoring in place, it should run nicely
> with very little intervention and let you be confident everything is as it
> should be.
>
> Key things to watch:
>
> * disk space - a disk filling up really makes things difficult for you.
> Make sure your retention fits your footprint.
> * consumer lag
> * isr (in sync replica) state - really, you just want to know about out of
> sync replicas
> * simple rates (produce/consume) to make sure things are sane.
> * standard up/down monitoring (though isr status will typically catch dead
> brokers)
>
> If you dont already have jmx monitoring tools, I suggest you look at
> jmxtrans and graphite/grafana, or elk stack. Both will enable you to
> monitor all the jmx stats available, and visualize easily, so you can
> hopefully just fire and (mostly) forget.
>
> We run about 100 brokers, moving 60billion+ messages a day, in a few
> different clusters, and aside from glancing at the dashboard occasionally,
> there is really no day-to-day maintenance required. Kafka just works.
>
> Probably our biggest operational overhead is when it comes time to roll a
> cluster to apply a change or do an upgrade. Waiting for everything to get
> back in sync and stable before starting another broker means it can take a
> couple days to complete an upgrade/change safely.
>
> Cheers
>
> Todd
>
> Sent from my BlackBerry 10 smartphone on the TELUS network.
> Original Message
> From: Kyle Mathews
> Sent: Sunday, February 14, 2016 7:07 PM
> To: users@kafka.apache.org
> Reply To: users@kafka.apache.org
> Subject: Kafka advice for small startup
>
>
> Hi I'm the technical co-founder of a startup in San Francisco and I've been
> investigating using Kafka for some time.
>
> I'm thinking about biting the bullet and integrating it soon into our
> system. My main concern is how much operational overhead it'll add. I'm the
> only engineer right now and am already stretched thin. Since Kafka isn't a
> must have yet, it only makes sense to adopt it if it's pretty care free.
>
> What are people's experiences in similar situations to mine? Is Kafka
> something you can setup and forget like say redis or is something I'll be
> babysitting a decent bit and spending a lot of time on upgrades etc.?
>
> Also I'd love to take anyone in San Francisco with Kafka experience out to
> lunch or coffee.
>
> Thanks!
> Kyle
>
--
--
Sharninder
Re: Kafka advice for small startup
Posted by Todd Snyder <ts...@blackberry.com>.
So long as you put some basic monitoring in place, it should run nicely with very little intervention and let you be confident everything is as it should be.
Key things to watch:
* disk space - a disk filling up really makes things difficult for you. Make sure your retention fits your footprint.
* consumer lag
* isr (in sync replica) state - really, you just want to know about out of sync replicas
* simple rates (produce/consume) to make sure things are sane.
* standard up/down monitoring (though isr status will typically catch dead brokers)
If you dont already have jmx monitoring tools, I suggest you look at jmxtrans and graphite/grafana, or elk stack. Both will enable you to monitor all the jmx stats available, and visualize easily, so you can hopefully just fire and (mostly) forget.
We run about 100 brokers, moving 60billion+ messages a day, in a few different clusters, and aside from glancing at the dashboard occasionally, there is really no day-to-day maintenance required. Kafka just works.
Probably our biggest operational overhead is when it comes time to roll a cluster to apply a change or do an upgrade. Waiting for everything to get back in sync and stable before starting another broker means it can take a couple days to complete an upgrade/change safely.
Cheers
Todd
Sent from my BlackBerry 10 smartphone on the TELUS network.
Original Message
From: Kyle Mathews
Sent: Sunday, February 14, 2016 7:07 PM
To: users@kafka.apache.org
Reply To: users@kafka.apache.org
Subject: Kafka advice for small startup
Hi I'm the technical co-founder of a startup in San Francisco and I've been
investigating using Kafka for some time.
I'm thinking about biting the bullet and integrating it soon into our
system. My main concern is how much operational overhead it'll add. I'm the
only engineer right now and am already stretched thin. Since Kafka isn't a
must have yet, it only makes sense to adopt it if it's pretty care free.
What are people's experiences in similar situations to mine? Is Kafka
something you can setup and forget like say redis or is something I'll be
babysitting a decent bit and spending a lot of time on upgrades etc.?
Also I'd love to take anyone in San Francisco with Kafka experience out to
lunch or coffee.
Thanks!
Kyle
Re: Kafka advice for small startup
Posted by ph...@yahoo.com.INVALID.
Just to be clear, Kafka is not something that needs babysitting. It just works (in the good way).
-----------------------------------------
http://www.philipotoole.com
On Sunday, February 14, 2016 4:17 PM, "philip.otoole@yahoo.com.INVALID" <ph...@yahoo.com.INVALID> wrote:
I've built both large and small systems with Kafka. It's not difficult to set up, and works like a charm. It should be the least of your worries, in my experience.
Philip
-----------------------------------------
http://www.philipotoole.com
On Sunday, February 14, 2016 4:07 PM, Kyle Mathews <ma...@gmail.com> wrote:
Hi I'm the technical co-founder of a startup in San Francisco and I've been
investigating using Kafka for some time.
I'm thinking about biting the bullet and integrating it soon into our
system. My main concern is how much operational overhead it'll add. I'm the
only engineer right now and am already stretched thin. Since Kafka isn't a
must have yet, it only makes sense to adopt it if it's pretty care free.
What are people's experiences in similar situations to mine? Is Kafka
something you can setup and forget like say redis or is something I'll be
babysitting a decent bit and spending a lot of time on upgrades etc.?
Also I'd love to take anyone in San Francisco with Kafka experience out to
lunch or coffee.
Thanks!
Kyle
Re: Kafka advice for small startup
Posted by ph...@yahoo.com.INVALID.
I've built both large and small systems with Kafka. It's not difficult to set up, and works like a charm. It should be the least of your worries, in my experience.
Philip
-----------------------------------------
http://www.philipotoole.com
On Sunday, February 14, 2016 4:07 PM, Kyle Mathews <ma...@gmail.com> wrote:
Hi I'm the technical co-founder of a startup in San Francisco and I've been
investigating using Kafka for some time.
I'm thinking about biting the bullet and integrating it soon into our
system. My main concern is how much operational overhead it'll add. I'm the
only engineer right now and am already stretched thin. Since Kafka isn't a
must have yet, it only makes sense to adopt it if it's pretty care free.
What are people's experiences in similar situations to mine? Is Kafka
something you can setup and forget like say redis or is something I'll be
babysitting a decent bit and spending a lot of time on upgrades etc.?
Also I'd love to take anyone in San Francisco with Kafka experience out to
lunch or coffee.
Thanks!
Kyle