Re: kafka + autoscaling groups fuckery

[Charity Majors <ch...@hound.sh>]

Gwen, thanks for the response.

1.1 Your life may be a bit simpler if you have a way of starting a new

> broker with the same ID as the old one - this means it will
> automatically pick up the old replicas and you won't need to
> rebalance. Makes life slightly easier in some cases.
>

Yeah, this is definitely doable, I just don't *want* to do it.  I really
want all of these to share the same code path: 1) rolling all nodes in an
ASG to pick up a new AMI, 2) hardware failure / unintentional node
termination, 3) resizing the ASG and rebalancing the data across nodes.

Everything but the first one means generating new node IDs, so I would
rather just do that across the board.  It's the solution that really fits
the ASG model best, so I'm reluctant to give up on it.


> 1.2 Careful not too rebalance too many partitions at once - you only
> have so much bandwidth and currently Kafka will not throttle
> rebalancing traffic.
>

Nod, got it.  This is def something I plan to work on hardening once I have
the basic nut of things working (or if I've had to give up on it and accept
a lesser solution).


> 2. I think your rebalance script is not rebalancing the offsets topic?
> It still has a replica on broker 1002. You have two good replicas, so
> you are no where near disaster, but make sure you get this working
> too.
>

Yes, this is another problem I am working on in parallel.  The Shopify
sarama library <https://godoc.org/github.com/Shopify/sarama> uses the
__consumer_offsets topic, but it does *not* let you rebalance or resize the
topic when consumers connect, disconnect, or restart.

"Note that Sarama's Consumer implementation does not currently support
automatic consumer-group rebalancing and offset tracking"

I'm working on trying to get the sarama-cluster to do something here.  I
think these problems are likely related, I'm not sure wtf you are
*supposed* to do to rebalance this god damn topic.  It also seems like we
aren't using a consumer group which sarama-cluster depends on to rebalance
a topic.  I'm still pretty confused by the 0.9 "consumer group" stuff.

Seriously considering downgrading to the latest 0.8 release, because
there's a massive gap in documentation for the new stuff in 0.9 (like
consumer groups) and we don't really need any of the new features.

A common work-around is to configure the consumer to handle "offset
> out of range" exception by jumping to the last offset available in the
> log. This is the behavior of the Java client, and it would have saved
> your consumer here. Go client looks very low level, so I don't know
> how easy it is to do that.
>

Erf, this seems like it would almost guarantee data loss.  :(  Will check
it out tho.

If I were you, I'd retest your ASG scripts without the auto leader
> election - since your own scripts can / should handle that.
>

Okay, this is straightforward enough.  Will try it.  And will keep tryingn
to figure out how to balance the __consumer_offsets topic, since I
increasingly think that's the key to this giant mess.

If anyone has any advice there, massively appreciated.

Thanks,

charity.

Re: kafka + autoscaling groups fuckery

[Gian Merlino <gi...@gmail.com>]

I think it'd be possible to avoid special-casing replacements, but it might
be a bad idea network-traffic-wise, especially for rolling upgrades.

My experience running Kafka on AWS is that rebalancing with multi-day
retention periods can take a really long time, and can torch the cluster if
you rebalance too much at once. Rebalancing the entire cluster at once was
a recipe for having the cluster DoS itself. So we chose to special-case
upgrades/replacements just to avoid having too much rebalancing traffic.
What we did was store all the Kafka data, including the broker id, on EBS
volumes. Replacements mean attaching an old EBS volume to a new instance –
that way the id and data are reused, and no rebalance traffic is needed
(just some catch-up). New additions mean creating a new EBS volume, new
broker id, and new instance, and balancing partitions onto it.

If you have short retention periods then this is probably less of an issue.

(NB: this experience is from Kafka 0.8.x, not sure if something has
improved in Kafka since then)

On Sun, Jul 3, 2016 at 11:35 AM, Charity Majors <ch...@hound.sh> wrote:

> It would be cool to know if Netflix runs kafka in ASGs ... I can't find any
> mention of it online.  (https://github.com/Netflix/suro/wiki/FAQ sorta
> implies that maybe they do, but it's not clear, and also old.)
>
> I've seen other people talking about running kafka in ASGs, e.g.
> http://blog.jimjh.com/building-elastic-clusters.html, but they all rely on
> reusing broker IDs.  Which certainly makes it easier, but imho is the wrong
> way to do this for all the reasons I listed before.
>
> On Sun, Jul 3, 2016 at 11:29 AM, Charity Majors <ch...@hound.sh> wrote:
>
> > Great talks, but not relevant to either of my problems -- the golang
> > client not rebalancing the consumer offset topic, or autoscaling group
> > behavior (which is I think is probably just a consequence of the first).
> >
> > Thanks though, there's good stuff in here.
> >
> > On Sun, Jul 3, 2016 at 10:23 AM, James Cheng <wu...@gmail.com>
> wrote:
> >
> >> Charity,
> >>
> >> I'm not sure about the specific problem you are having, but about Kafka
> >> on AWS, Netflix did a talk at a meetup about their Kafka installation on
> >> AWS. There might be some useful information in there. There is a video
> >> stream as well as slides, and maybe you can get in touch with the
> speakers.
> >> Look in the comment section for links to the slides and video.
> >>
> >> Kafka at Netflix
> >>
> >>
> http://www.meetup.com//http-kafka-apache-org/events/220355031/?showDescription=true
> >>
> >> There's also a talk about running Kafka on Mesos, which might be
> relevant.
> >>
> >> Kafka on Mesos
> >>
> >>
> http://www.meetup.com//http-kafka-apache-org/events/222537743/?showDescription=true
> >>
> >> -James
> >>
> >> Sent from my iPhone
> >>
> >> > On Jul 2, 2016, at 5:15 PM, Charity Majors <ch...@hound.sh> wrote:
> >> >
> >> > Gwen, thanks for the response.
> >> >
> >> > 1.1 Your life may be a bit simpler if you have a way of starting a new
> >> >
> >> >> broker with the same ID as the old one - this means it will
> >> >> automatically pick up the old replicas and you won't need to
> >> >> rebalance. Makes life slightly easier in some cases.
> >> >
> >> > Yeah, this is definitely doable, I just don't *want* to do it.  I
> really
> >> > want all of these to share the same code path: 1) rolling all nodes in
> >> an
> >> > ASG to pick up a new AMI, 2) hardware failure / unintentional node
> >> > termination, 3) resizing the ASG and rebalancing the data across
> nodes.
> >> >
> >> > Everything but the first one means generating new node IDs, so I would
> >> > rather just do that across the board.  It's the solution that really
> >> fits
> >> > the ASG model best, so I'm reluctant to give up on it.
> >> >
> >> >
> >> >> 1.2 Careful not too rebalance too many partitions at once - you only
> >> >> have so much bandwidth and currently Kafka will not throttle
> >> >> rebalancing traffic.
> >> >
> >> > Nod, got it.  This is def something I plan to work on hardening once I
> >> have
> >> > the basic nut of things working (or if I've had to give up on it and
> >> accept
> >> > a lesser solution).
> >> >
> >> >
> >> >> 2. I think your rebalance script is not rebalancing the offsets
> topic?
> >> >> It still has a replica on broker 1002. You have two good replicas, so
> >> >> you are no where near disaster, but make sure you get this working
> >> >> too.
> >> >
> >> > Yes, this is another problem I am working on in parallel.  The Shopify
> >> > sarama library <https://godoc.org/github.com/Shopify/sarama> uses the
> >> > __consumer_offsets topic, but it does *not* let you rebalance or
> resize
> >> the
> >> > topic when consumers connect, disconnect, or restart.
> >> >
> >> > "Note that Sarama's Consumer implementation does not currently support
> >> > automatic consumer-group rebalancing and offset tracking"
> >> >
> >> > I'm working on trying to get the sarama-cluster to do something
> here.  I
> >> > think these problems are likely related, I'm not sure wtf you are
> >> > *supposed* to do to rebalance this god damn topic.  It also seems like
> >> we
> >> > aren't using a consumer group which sarama-cluster depends on to
> >> rebalance
> >> > a topic.  I'm still pretty confused by the 0.9 "consumer group" stuff.
> >> >
> >> > Seriously considering downgrading to the latest 0.8 release, because
> >> > there's a massive gap in documentation for the new stuff in 0.9 (like
> >> > consumer groups) and we don't really need any of the new features.
> >> >
> >> > A common work-around is to configure the consumer to handle "offset
> >> >> out of range" exception by jumping to the last offset available in
> the
> >> >> log. This is the behavior of the Java client, and it would have saved
> >> >> your consumer here. Go client looks very low level, so I don't know
> >> >> how easy it is to do that.
> >> >
> >> > Erf, this seems like it would almost guarantee data loss.  :(  Will
> >> check
> >> > it out tho.
> >> >
> >> > If I were you, I'd retest your ASG scripts without the auto leader
> >> >> election - since your own scripts can / should handle that.
> >> >
> >> > Okay, this is straightforward enough.  Will try it.  And will keep
> >> tryingn
> >> > to figure out how to balance the __consumer_offsets topic, since I
> >> > increasingly think that's the key to this giant mess.
> >> >
> >> > If anyone has any advice there, massively appreciated.
> >> >
> >> > Thanks,
> >> >
> >> > charity.
> >>
> >
> >
>

Re: kafka + autoscaling groups fuckery

[Charity Majors <ch...@hound.sh>]

It would be cool to know if Netflix runs kafka in ASGs ... I can't find any
mention of it online.  (https://github.com/Netflix/suro/wiki/FAQ sorta
implies that maybe they do, but it's not clear, and also old.)

I've seen other people talking about running kafka in ASGs, e.g.
http://blog.jimjh.com/building-elastic-clusters.html, but they all rely on
reusing broker IDs.  Which certainly makes it easier, but imho is the wrong
way to do this for all the reasons I listed before.

On Sun, Jul 3, 2016 at 11:29 AM, Charity Majors <ch...@hound.sh> wrote:

> Great talks, but not relevant to either of my problems -- the golang
> client not rebalancing the consumer offset topic, or autoscaling group
> behavior (which is I think is probably just a consequence of the first).
>
> Thanks though, there's good stuff in here.
>
> On Sun, Jul 3, 2016 at 10:23 AM, James Cheng <wu...@gmail.com> wrote:
>
>> Charity,
>>
>> I'm not sure about the specific problem you are having, but about Kafka
>> on AWS, Netflix did a talk at a meetup about their Kafka installation on
>> AWS. There might be some useful information in there. There is a video
>> stream as well as slides, and maybe you can get in touch with the speakers.
>> Look in the comment section for links to the slides and video.
>>
>> Kafka at Netflix
>>
>> http://www.meetup.com//http-kafka-apache-org/events/220355031/?showDescription=true
>>
>> There's also a talk about running Kafka on Mesos, which might be relevant.
>>
>> Kafka on Mesos
>>
>> http://www.meetup.com//http-kafka-apache-org/events/222537743/?showDescription=true
>>
>> -James
>>
>> Sent from my iPhone
>>
>> > On Jul 2, 2016, at 5:15 PM, Charity Majors <ch...@hound.sh> wrote:
>> >
>> > Gwen, thanks for the response.
>> >
>> > 1.1 Your life may be a bit simpler if you have a way of starting a new
>> >
>> >> broker with the same ID as the old one - this means it will
>> >> automatically pick up the old replicas and you won't need to
>> >> rebalance. Makes life slightly easier in some cases.
>> >
>> > Yeah, this is definitely doable, I just don't *want* to do it.  I really
>> > want all of these to share the same code path: 1) rolling all nodes in
>> an
>> > ASG to pick up a new AMI, 2) hardware failure / unintentional node
>> > termination, 3) resizing the ASG and rebalancing the data across nodes.
>> >
>> > Everything but the first one means generating new node IDs, so I would
>> > rather just do that across the board.  It's the solution that really
>> fits
>> > the ASG model best, so I'm reluctant to give up on it.
>> >
>> >
>> >> 1.2 Careful not too rebalance too many partitions at once - you only
>> >> have so much bandwidth and currently Kafka will not throttle
>> >> rebalancing traffic.
>> >
>> > Nod, got it.  This is def something I plan to work on hardening once I
>> have
>> > the basic nut of things working (or if I've had to give up on it and
>> accept
>> > a lesser solution).
>> >
>> >
>> >> 2. I think your rebalance script is not rebalancing the offsets topic?
>> >> It still has a replica on broker 1002. You have two good replicas, so
>> >> you are no where near disaster, but make sure you get this working
>> >> too.
>> >
>> > Yes, this is another problem I am working on in parallel.  The Shopify
>> > sarama library <https://godoc.org/github.com/Shopify/sarama> uses the
>> > __consumer_offsets topic, but it does *not* let you rebalance or resize
>> the
>> > topic when consumers connect, disconnect, or restart.
>> >
>> > "Note that Sarama's Consumer implementation does not currently support
>> > automatic consumer-group rebalancing and offset tracking"
>> >
>> > I'm working on trying to get the sarama-cluster to do something here.  I
>> > think these problems are likely related, I'm not sure wtf you are
>> > *supposed* to do to rebalance this god damn topic.  It also seems like
>> we
>> > aren't using a consumer group which sarama-cluster depends on to
>> rebalance
>> > a topic.  I'm still pretty confused by the 0.9 "consumer group" stuff.
>> >
>> > Seriously considering downgrading to the latest 0.8 release, because
>> > there's a massive gap in documentation for the new stuff in 0.9 (like
>> > consumer groups) and we don't really need any of the new features.
>> >
>> > A common work-around is to configure the consumer to handle "offset
>> >> out of range" exception by jumping to the last offset available in the
>> >> log. This is the behavior of the Java client, and it would have saved
>> >> your consumer here. Go client looks very low level, so I don't know
>> >> how easy it is to do that.
>> >
>> > Erf, this seems like it would almost guarantee data loss.  :(  Will
>> check
>> > it out tho.
>> >
>> > If I were you, I'd retest your ASG scripts without the auto leader
>> >> election - since your own scripts can / should handle that.
>> >
>> > Okay, this is straightforward enough.  Will try it.  And will keep
>> tryingn
>> > to figure out how to balance the __consumer_offsets topic, since I
>> > increasingly think that's the key to this giant mess.
>> >
>> > If anyone has any advice there, massively appreciated.
>> >
>> > Thanks,
>> >
>> > charity.
>>
>
>

Re: kafka + autoscaling groups fuckery

[Charity Majors <ch...@hound.sh>]

Great talks, but not relevant to either of my problems -- the golang client
not rebalancing the consumer offset topic, or autoscaling group behavior
(which is I think is probably just a consequence of the first).

Thanks though, there's good stuff in here.

On Sun, Jul 3, 2016 at 10:23 AM, James Cheng <wu...@gmail.com> wrote:

> Charity,
>
> I'm not sure about the specific problem you are having, but about Kafka on
> AWS, Netflix did a talk at a meetup about their Kafka installation on AWS.
> There might be some useful information in there. There is a video stream as
> well as slides, and maybe you can get in touch with the speakers. Look in
> the comment section for links to the slides and video.
>
> Kafka at Netflix
>
> http://www.meetup.com//http-kafka-apache-org/events/220355031/?showDescription=true
>
> There's also a talk about running Kafka on Mesos, which might be relevant.
>
> Kafka on Mesos
>
> http://www.meetup.com//http-kafka-apache-org/events/222537743/?showDescription=true
>
> -James
>
> Sent from my iPhone
>
> > On Jul 2, 2016, at 5:15 PM, Charity Majors <ch...@hound.sh> wrote:
> >
> > Gwen, thanks for the response.
> >
> > 1.1 Your life may be a bit simpler if you have a way of starting a new
> >
> >> broker with the same ID as the old one - this means it will
> >> automatically pick up the old replicas and you won't need to
> >> rebalance. Makes life slightly easier in some cases.
> >
> > Yeah, this is definitely doable, I just don't *want* to do it.  I really
> > want all of these to share the same code path: 1) rolling all nodes in an
> > ASG to pick up a new AMI, 2) hardware failure / unintentional node
> > termination, 3) resizing the ASG and rebalancing the data across nodes.
> >
> > Everything but the first one means generating new node IDs, so I would
> > rather just do that across the board.  It's the solution that really fits
> > the ASG model best, so I'm reluctant to give up on it.
> >
> >
> >> 1.2 Careful not too rebalance too many partitions at once - you only
> >> have so much bandwidth and currently Kafka will not throttle
> >> rebalancing traffic.
> >
> > Nod, got it.  This is def something I plan to work on hardening once I
> have
> > the basic nut of things working (or if I've had to give up on it and
> accept
> > a lesser solution).
> >
> >
> >> 2. I think your rebalance script is not rebalancing the offsets topic?
> >> It still has a replica on broker 1002. You have two good replicas, so
> >> you are no where near disaster, but make sure you get this working
> >> too.
> >
> > Yes, this is another problem I am working on in parallel.  The Shopify
> > sarama library <https://godoc.org/github.com/Shopify/sarama> uses the
> > __consumer_offsets topic, but it does *not* let you rebalance or resize
> the
> > topic when consumers connect, disconnect, or restart.
> >
> > "Note that Sarama's Consumer implementation does not currently support
> > automatic consumer-group rebalancing and offset tracking"
> >
> > I'm working on trying to get the sarama-cluster to do something here.  I
> > think these problems are likely related, I'm not sure wtf you are
> > *supposed* to do to rebalance this god damn topic.  It also seems like we
> > aren't using a consumer group which sarama-cluster depends on to
> rebalance
> > a topic.  I'm still pretty confused by the 0.9 "consumer group" stuff.
> >
> > Seriously considering downgrading to the latest 0.8 release, because
> > there's a massive gap in documentation for the new stuff in 0.9 (like
> > consumer groups) and we don't really need any of the new features.
> >
> > A common work-around is to configure the consumer to handle "offset
> >> out of range" exception by jumping to the last offset available in the
> >> log. This is the behavior of the Java client, and it would have saved
> >> your consumer here. Go client looks very low level, so I don't know
> >> how easy it is to do that.
> >
> > Erf, this seems like it would almost guarantee data loss.  :(  Will check
> > it out tho.
> >
> > If I were you, I'd retest your ASG scripts without the auto leader
> >> election - since your own scripts can / should handle that.
> >
> > Okay, this is straightforward enough.  Will try it.  And will keep
> tryingn
> > to figure out how to balance the __consumer_offsets topic, since I
> > increasingly think that's the key to this giant mess.
> >
> > If anyone has any advice there, massively appreciated.
> >
> > Thanks,
> >
> > charity.
>

Re: kafka + autoscaling groups fuckery

[James Cheng <wu...@gmail.com>]

Charity,

I'm not sure about the specific problem you are having, but about Kafka on AWS, Netflix did a talk at a meetup about their Kafka installation on AWS. There might be some useful information in there. There is a video stream as well as slides, and maybe you can get in touch with the speakers. Look in the comment section for links to the slides and video. 

Kafka at Netflix
http://www.meetup.com//http-kafka-apache-org/events/220355031/?showDescription=true

There's also a talk about running Kafka on Mesos, which might be relevant.

Kafka on Mesos
http://www.meetup.com//http-kafka-apache-org/events/222537743/?showDescription=true

-James

Sent from my iPhone

> On Jul 2, 2016, at 5:15 PM, Charity Majors <ch...@hound.sh> wrote:
> 
> Gwen, thanks for the response.
> 
> 1.1 Your life may be a bit simpler if you have a way of starting a new
> 
>> broker with the same ID as the old one - this means it will
>> automatically pick up the old replicas and you won't need to
>> rebalance. Makes life slightly easier in some cases.
> 
> Yeah, this is definitely doable, I just don't *want* to do it.  I really
> want all of these to share the same code path: 1) rolling all nodes in an
> ASG to pick up a new AMI, 2) hardware failure / unintentional node
> termination, 3) resizing the ASG and rebalancing the data across nodes.
> 
> Everything but the first one means generating new node IDs, so I would
> rather just do that across the board.  It's the solution that really fits
> the ASG model best, so I'm reluctant to give up on it.
> 
> 
>> 1.2 Careful not too rebalance too many partitions at once - you only
>> have so much bandwidth and currently Kafka will not throttle
>> rebalancing traffic.
> 
> Nod, got it.  This is def something I plan to work on hardening once I have
> the basic nut of things working (or if I've had to give up on it and accept
> a lesser solution).
> 
> 
>> 2. I think your rebalance script is not rebalancing the offsets topic?
>> It still has a replica on broker 1002. You have two good replicas, so
>> you are no where near disaster, but make sure you get this working
>> too.
> 
> Yes, this is another problem I am working on in parallel.  The Shopify
> sarama library <https://godoc.org/github.com/Shopify/sarama> uses the
> __consumer_offsets topic, but it does *not* let you rebalance or resize the
> topic when consumers connect, disconnect, or restart.
> 
> "Note that Sarama's Consumer implementation does not currently support
> automatic consumer-group rebalancing and offset tracking"
> 
> I'm working on trying to get the sarama-cluster to do something here.  I
> think these problems are likely related, I'm not sure wtf you are
> *supposed* to do to rebalance this god damn topic.  It also seems like we
> aren't using a consumer group which sarama-cluster depends on to rebalance
> a topic.  I'm still pretty confused by the 0.9 "consumer group" stuff.
> 
> Seriously considering downgrading to the latest 0.8 release, because
> there's a massive gap in documentation for the new stuff in 0.9 (like
> consumer groups) and we don't really need any of the new features.
> 
> A common work-around is to configure the consumer to handle "offset
>> out of range" exception by jumping to the last offset available in the
>> log. This is the behavior of the Java client, and it would have saved
>> your consumer here. Go client looks very low level, so I don't know
>> how easy it is to do that.
> 
> Erf, this seems like it would almost guarantee data loss.  :(  Will check
> it out tho.
> 
> If I were you, I'd retest your ASG scripts without the auto leader
>> election - since your own scripts can / should handle that.
> 
> Okay, this is straightforward enough.  Will try it.  And will keep tryingn
> to figure out how to balance the __consumer_offsets topic, since I
> increasingly think that's the key to this giant mess.
> 
> If anyone has any advice there, massively appreciated.
> 
> Thanks,
> 
> charity.