You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Ben Stopford <be...@confluent.io> on 2017/01/03 18:54:36 UTC
[DISCUSS] KIP-106 - Change Default unclean.leader.election.enabled
from True to False
Hi All
Please find the below KIP which proposes changing the setting unclean.leader.election.enabled from true to false. The motivation for this change is that it catches out new Kafka users who don’t realise the default favours availability over data loss.
This would mean clusters wishing to continue with unclean leader election enabled would need to add the appropriate configuration on upgrade.
Please let me know if you foresee any issue with this change, agree or don’t agree.
https://cwiki.apache.org/confluence/display/KAFKA/%5BWIP%5D+KIP-106+-+Change+Default+unclean.leader.election.enabled+from+True+to+False <https://cwiki.apache.org/confluence/display/KAFKA/[WIP]+KIP-106+-+Change+Default+unclean.leader.election.enabled+from+True+to+False>
Thanks
B
Ben Stopford
Confluent, http://www.confluent.io <http://www.confluent.io/>
Re: [DISCUSS] KIP-106 - Change Default unclean.leader.election.enabled
from True to False
Posted by Gwen Shapira <gw...@confluent.io>.
I strongly support this. I've seen many users and customers
accidentally lose data because they didn't know this configuration
exists. I always support defaulting to "Not lose data" and let the
users who prefer high availability do some extra research. This will
need to be very well documented - both the default trade-off, the
alternative and the fact that things changed.
On Tue, Jan 3, 2017 at 10:54 AM, Ben Stopford <be...@confluent.io> wrote:
> Hi All
>
> Please find the below KIP which proposes changing the setting unclean.leader.election.enabled from true to false. The motivation for this change is that it catches out new Kafka users who don’t realise the default favours availability over data loss.
>
> This would mean clusters wishing to continue with unclean leader election enabled would need to add the appropriate configuration on upgrade.
>
> Please let me know if you foresee any issue with this change, agree or don’t agree.
>
> https://cwiki.apache.org/confluence/display/KAFKA/%5BWIP%5D+KIP-106+-+Change+Default+unclean.leader.election.enabled+from+True+to+False <https://cwiki.apache.org/confluence/display/KAFKA/[WIP]+KIP-106+-+Change+Default+unclean.leader.election.enabled+from+True+to+False>
>
> Thanks
>
> B
>
> Ben Stopford
> Confluent, http://www.confluent.io <http://www.confluent.io/>
>
>
>
--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog
Re: [DISCUSS] KIP-106 - Change Default unclean.leader.election.enabled
from True to False
Posted by Onur Karaman <ok...@linkedin.com.INVALID>.
+1
On Tue, Jan 3, 2017 at 1:12 PM, Sriram Subramanian <ra...@confluent.io> wrote:
> +1
>
> On Tue, Jan 3, 2017 at 11:56 AM, Ian Wrigley <ia...@confluent.io> wrote:
>
> > +1 from me too. When I talk to people in training classes, who are
> > typically much newer to Kafka, they tend to be surprised (and
> > scared/horrified) that the default is true. Much safer to set it to false
> > and let people change it when they really understand the tradeoffs.
> >
> > Ian.
> >
> > ---
> > Ian Wrigley
> > Director, Education Services
> > Confluent, Inc
> >
> > > On Jan 3, 2017, at 1:22 PM, Tom Crayford <tc...@heroku.com> wrote:
> > >
> > > +1. We've been running it in production for a long time and it's the
> > right
> > > default.
> > >
> > > On Tue, Jan 3, 2017 at 7:17 PM, Ismael Juma <is...@juma.me.uk> wrote:
> > >
> > >> Thanks for the KIP, +1 from me.
> > >>
> > >> Ismael
> > >>
> > >> On 3 Jan 2017 6:54 pm, "Ben Stopford" <be...@confluent.io> wrote:
> > >>
> > >>> Hi All
> > >>>
> > >>> Please find the below KIP which proposes changing the setting
> > >>> unclean.leader.election.enabled from true to false. The motivation
> for
> > >>> this change is that it catches out new Kafka users who don’t realise
> > the
> > >>> default favours availability over data loss.
> > >>>
> > >>> This would mean clusters wishing to continue with unclean leader
> > election
> > >>> enabled would need to add the appropriate configuration on upgrade.
> > >>>
> > >>> Please let me know if you foresee any issue with this change, agree
> or
> > >>> don’t agree.
> > >>>
> > >>> https://cwiki.apache.org/confluence/display/KAFKA/%
> > >>> 5BWIP%5D+KIP-106+-+Change+Default+unclean.leader.
> > >>> election.enabled+from+True+to+False <https://cwiki.apache.org/
> > >>> confluence/display/KAFKA/[WIP]+KIP-106+-+Change+Default+
> > >>> unclean.leader.election.enabled+from+True+to+False>
> > >>>
> > >>> Thanks
> > >>>
> > >>> B
> > >>>
> > >>> Ben Stopford
> > >>> Confluent, http://www.confluent.io <http://www.confluent.io/>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> >
>
Re: [DISCUSS] KIP-106 - Change Default unclean.leader.election.enabled
from True to False
Posted by Sriram Subramanian <ra...@confluent.io>.
+1
On Tue, Jan 3, 2017 at 11:56 AM, Ian Wrigley <ia...@confluent.io> wrote:
> +1 from me too. When I talk to people in training classes, who are
> typically much newer to Kafka, they tend to be surprised (and
> scared/horrified) that the default is true. Much safer to set it to false
> and let people change it when they really understand the tradeoffs.
>
> Ian.
>
> ---
> Ian Wrigley
> Director, Education Services
> Confluent, Inc
>
> > On Jan 3, 2017, at 1:22 PM, Tom Crayford <tc...@heroku.com> wrote:
> >
> > +1. We've been running it in production for a long time and it's the
> right
> > default.
> >
> > On Tue, Jan 3, 2017 at 7:17 PM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> >> Thanks for the KIP, +1 from me.
> >>
> >> Ismael
> >>
> >> On 3 Jan 2017 6:54 pm, "Ben Stopford" <be...@confluent.io> wrote:
> >>
> >>> Hi All
> >>>
> >>> Please find the below KIP which proposes changing the setting
> >>> unclean.leader.election.enabled from true to false. The motivation for
> >>> this change is that it catches out new Kafka users who don’t realise
> the
> >>> default favours availability over data loss.
> >>>
> >>> This would mean clusters wishing to continue with unclean leader
> election
> >>> enabled would need to add the appropriate configuration on upgrade.
> >>>
> >>> Please let me know if you foresee any issue with this change, agree or
> >>> don’t agree.
> >>>
> >>> https://cwiki.apache.org/confluence/display/KAFKA/%
> >>> 5BWIP%5D+KIP-106+-+Change+Default+unclean.leader.
> >>> election.enabled+from+True+to+False <https://cwiki.apache.org/
> >>> confluence/display/KAFKA/[WIP]+KIP-106+-+Change+Default+
> >>> unclean.leader.election.enabled+from+True+to+False>
> >>>
> >>> Thanks
> >>>
> >>> B
> >>>
> >>> Ben Stopford
> >>> Confluent, http://www.confluent.io <http://www.confluent.io/>
> >>>
> >>>
> >>>
> >>>
> >>
>
>
Re: [DISCUSS] KIP-106 - Change Default
unclean.leader.election.enabled from True to False
Posted by Ian Wrigley <ia...@confluent.io>.
+1 from me too. When I talk to people in training classes, who are typically much newer to Kafka, they tend to be surprised (and scared/horrified) that the default is true. Much safer to set it to false and let people change it when they really understand the tradeoffs.
Ian.
---
Ian Wrigley
Director, Education Services
Confluent, Inc
> On Jan 3, 2017, at 1:22 PM, Tom Crayford <tc...@heroku.com> wrote:
>
> +1. We've been running it in production for a long time and it's the right
> default.
>
> On Tue, Jan 3, 2017 at 7:17 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
>> Thanks for the KIP, +1 from me.
>>
>> Ismael
>>
>> On 3 Jan 2017 6:54 pm, "Ben Stopford" <be...@confluent.io> wrote:
>>
>>> Hi All
>>>
>>> Please find the below KIP which proposes changing the setting
>>> unclean.leader.election.enabled from true to false. The motivation for
>>> this change is that it catches out new Kafka users who don’t realise the
>>> default favours availability over data loss.
>>>
>>> This would mean clusters wishing to continue with unclean leader election
>>> enabled would need to add the appropriate configuration on upgrade.
>>>
>>> Please let me know if you foresee any issue with this change, agree or
>>> don’t agree.
>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/%
>>> 5BWIP%5D+KIP-106+-+Change+Default+unclean.leader.
>>> election.enabled+from+True+to+False <https://cwiki.apache.org/
>>> confluence/display/KAFKA/[WIP]+KIP-106+-+Change+Default+
>>> unclean.leader.election.enabled+from+True+to+False>
>>>
>>> Thanks
>>>
>>> B
>>>
>>> Ben Stopford
>>> Confluent, http://www.confluent.io <http://www.confluent.io/>
>>>
>>>
>>>
>>>
>>
Re: [DISCUSS] KIP-106 - Change Default unclean.leader.election.enabled
from True to False
Posted by Neha Narkhede <ne...@confluent.io>.
Big +1.
On Tue, Jan 3, 2017 at 11:22 AM Tom Crayford <tc...@heroku.com> wrote:
> +1. We've been running it in production for a long time and it's the right
> default.
>
> On Tue, Jan 3, 2017 at 7:17 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Thanks for the KIP, +1 from me.
> >
> > Ismael
> >
> > On 3 Jan 2017 6:54 pm, "Ben Stopford" <be...@confluent.io> wrote:
> >
> > > Hi All
> > >
> > > Please find the below KIP which proposes changing the setting
> > > unclean.leader.election.enabled from true to false. The motivation for
> > > this change is that it catches out new Kafka users who don’t realise
> the
> > > default favours availability over data loss.
> > >
> > > This would mean clusters wishing to continue with unclean leader
> election
> > > enabled would need to add the appropriate configuration on upgrade.
> > >
> > > Please let me know if you foresee any issue with this change, agree or
> > > don’t agree.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/%
> > > 5BWIP%5D+KIP-106+-+Change+Default+unclean.leader.
> > > election.enabled+from+True+to+False <https://cwiki.apache.org/
> > > confluence/display/KAFKA/[WIP]+KIP-106+-+Change+Default+
> > > unclean.leader.election.enabled+from+True+to+False>
> > >
> > > Thanks
> > >
> > > B
> > >
> > > Ben Stopford
> > > Confluent, http://www.confluent.io <http://www.confluent.io/>
> > >
> > >
> > >
> > >
> >
>
--
Thanks,
Neha
Re: [DISCUSS] KIP-106 - Change Default unclean.leader.election.enabled
from True to False
Posted by Tom Crayford <tc...@heroku.com>.
+1. We've been running it in production for a long time and it's the right
default.
On Tue, Jan 3, 2017 at 7:17 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Thanks for the KIP, +1 from me.
>
> Ismael
>
> On 3 Jan 2017 6:54 pm, "Ben Stopford" <be...@confluent.io> wrote:
>
> > Hi All
> >
> > Please find the below KIP which proposes changing the setting
> > unclean.leader.election.enabled from true to false. The motivation for
> > this change is that it catches out new Kafka users who don’t realise the
> > default favours availability over data loss.
> >
> > This would mean clusters wishing to continue with unclean leader election
> > enabled would need to add the appropriate configuration on upgrade.
> >
> > Please let me know if you foresee any issue with this change, agree or
> > don’t agree.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/%
> > 5BWIP%5D+KIP-106+-+Change+Default+unclean.leader.
> > election.enabled+from+True+to+False <https://cwiki.apache.org/
> > confluence/display/KAFKA/[WIP]+KIP-106+-+Change+Default+
> > unclean.leader.election.enabled+from+True+to+False>
> >
> > Thanks
> >
> > B
> >
> > Ben Stopford
> > Confluent, http://www.confluent.io <http://www.confluent.io/>
> >
> >
> >
> >
>
Re: [DISCUSS] KIP-106 - Change Default unclean.leader.election.enabled
from True to False
Posted by Ismael Juma <is...@juma.me.uk>.
Thanks for the KIP, +1 from me.
Ismael
On 3 Jan 2017 6:54 pm, "Ben Stopford" <be...@confluent.io> wrote:
> Hi All
>
> Please find the below KIP which proposes changing the setting
> unclean.leader.election.enabled from true to false. The motivation for
> this change is that it catches out new Kafka users who don’t realise the
> default favours availability over data loss.
>
> This would mean clusters wishing to continue with unclean leader election
> enabled would need to add the appropriate configuration on upgrade.
>
> Please let me know if you foresee any issue with this change, agree or
> don’t agree.
>
> https://cwiki.apache.org/confluence/display/KAFKA/%
> 5BWIP%5D+KIP-106+-+Change+Default+unclean.leader.
> election.enabled+from+True+to+False <https://cwiki.apache.org/
> confluence/display/KAFKA/[WIP]+KIP-106+-+Change+Default+
> unclean.leader.election.enabled+from+True+to+False>
>
> Thanks
>
> B
>
> Ben Stopford
> Confluent, http://www.confluent.io <http://www.confluent.io/>
>
>
>
>
Re: [DISCUSS] KIP-106 - Change Default
unclean.leader.election.enabled from True to False
Posted by James Cheng <wu...@gmail.com>.
+1
I agree with Gwen that this needs to be well communicated to users. Anyone who is using the current default may see their brokers start to go exit when they upgrade to this new version.
On a somewhat related subject, occasionally I'll need to look up how my Kafka brokers are configured. Typically, I will either
1) Look at the broker's server.log to see what is printed out at startup
OR
2a) look in the broker's config file for explicit overrides
2b) look in at the online docs to figure out defaults
2c) Do some not-that-hard mental layering to figure out which value is the right one.
Option 1 may not be available, if the server logs have been rotated out.
Option 2 is a multi-step process.
I wonder if it might be useful to have a "dry-run" mode where Kafka would parse its config file, calculate its config (using the defaults and overrides), and then print out the result and exit. This would help with detecting changes in config values from one version to another, without requiring actually launching the broker or closely tracking the docs.
And/or, a way to query the running broker and have it dump out its config on demand.
The same type of thing also applies to topic configuration. ./bin/kafka-configs.sh --describe can be used to look up overrides, but I still have to cross reference the broker config (by doing the above) to figure out how a topic is actually configured.
Users could then use that functionality to quickly get an idea of how a new version's configuration has changed, compared to what they are currently running.
I know such a thing is out of scope for this KIP. I just wanted to mention it, because it's related to changing of default values (and being aware of changes).
-James
> On Jan 4, 2017, at 11:51 AM, Rajini Sivaram <ra...@gmail.com> wrote:
>
> +1
>
> On Wed, Jan 4, 2017 at 1:49 AM, Becket Qin <be...@gmail.com> wrote:
>
>> +1
>>
>> On Tue, Jan 3, 2017 at 5:41 PM, Joel Koshy <jj...@gmail.com> wrote:
>>
>>> +1
>>>
>>> On Tue, Jan 3, 2017 at 10:54 AM, Ben Stopford <be...@confluent.io> wrote:
>>>
>>>> Hi All
>>>>
>>>> Please find the below KIP which proposes changing the setting
>>>> unclean.leader.election.enabled from true to false. The motivation for
>>>> this change is that it catches out new Kafka users who don’t realise
>> the
>>>> default favours availability over data loss.
>>>>
>>>> This would mean clusters wishing to continue with unclean leader
>> election
>>>> enabled would need to add the appropriate configuration on upgrade.
>>>>
>>>> Please let me know if you foresee any issue with this change, agree or
>>>> don’t agree.
>>>>
>>>> https://cwiki.apache.org/confluence/display/KAFKA/%5BWIP%5D+
>>>> KIP-106+-+Change+Default+unclean.leader.election.
>>>> enabled+from+True+to+False <https://cwiki.apache.org/conf
>>>> luence/display/KAFKA/[WIP]+KIP-106+-+Change+Default+uncle
>>>> an.leader.election.enabled+from+True+to+False
>>>> <https://cwiki.apache.org/confluence/display/KAFKA/%
>>> 5BWIP%5D+KIP-106+-+Change+Default+unclean.leader.
>>> election.enabled+from+True+to+False>
>>>>>
>>>>
>>>> Thanks
>>>>
>>>> B
>>>>
>>>> Ben Stopford
>>>> Confluent, http://www.confluent.io <http://www.confluent.io/>
>>>>
>>>>
>>>>
>>>>
>>>
>>
Re: [DISCUSS] KIP-106 - Change Default unclean.leader.election.enabled
from True to False
Posted by Rajini Sivaram <ra...@gmail.com>.
+1
On Wed, Jan 4, 2017 at 1:49 AM, Becket Qin <be...@gmail.com> wrote:
> +1
>
> On Tue, Jan 3, 2017 at 5:41 PM, Joel Koshy <jj...@gmail.com> wrote:
>
> > +1
> >
> > On Tue, Jan 3, 2017 at 10:54 AM, Ben Stopford <be...@confluent.io> wrote:
> >
> > > Hi All
> > >
> > > Please find the below KIP which proposes changing the setting
> > > unclean.leader.election.enabled from true to false. The motivation for
> > > this change is that it catches out new Kafka users who don’t realise
> the
> > > default favours availability over data loss.
> > >
> > > This would mean clusters wishing to continue with unclean leader
> election
> > > enabled would need to add the appropriate configuration on upgrade.
> > >
> > > Please let me know if you foresee any issue with this change, agree or
> > > don’t agree.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/%5BWIP%5D+
> > > KIP-106+-+Change+Default+unclean.leader.election.
> > > enabled+from+True+to+False <https://cwiki.apache.org/conf
> > > luence/display/KAFKA/[WIP]+KIP-106+-+Change+Default+uncle
> > > an.leader.election.enabled+from+True+to+False
> > > <https://cwiki.apache.org/confluence/display/KAFKA/%
> > 5BWIP%5D+KIP-106+-+Change+Default+unclean.leader.
> > election.enabled+from+True+to+False>
> > > >
> > >
> > > Thanks
> > >
> > > B
> > >
> > > Ben Stopford
> > > Confluent, http://www.confluent.io <http://www.confluent.io/>
> > >
> > >
> > >
> > >
> >
>
Re: [DISCUSS] KIP-106 - Change Default unclean.leader.election.enabled
from True to False
Posted by Becket Qin <be...@gmail.com>.
+1
On Tue, Jan 3, 2017 at 5:41 PM, Joel Koshy <jj...@gmail.com> wrote:
> +1
>
> On Tue, Jan 3, 2017 at 10:54 AM, Ben Stopford <be...@confluent.io> wrote:
>
> > Hi All
> >
> > Please find the below KIP which proposes changing the setting
> > unclean.leader.election.enabled from true to false. The motivation for
> > this change is that it catches out new Kafka users who don’t realise the
> > default favours availability over data loss.
> >
> > This would mean clusters wishing to continue with unclean leader election
> > enabled would need to add the appropriate configuration on upgrade.
> >
> > Please let me know if you foresee any issue with this change, agree or
> > don’t agree.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/%5BWIP%5D+
> > KIP-106+-+Change+Default+unclean.leader.election.
> > enabled+from+True+to+False <https://cwiki.apache.org/conf
> > luence/display/KAFKA/[WIP]+KIP-106+-+Change+Default+uncle
> > an.leader.election.enabled+from+True+to+False
> > <https://cwiki.apache.org/confluence/display/KAFKA/%
> 5BWIP%5D+KIP-106+-+Change+Default+unclean.leader.
> election.enabled+from+True+to+False>
> > >
> >
> > Thanks
> >
> > B
> >
> > Ben Stopford
> > Confluent, http://www.confluent.io <http://www.confluent.io/>
> >
> >
> >
> >
>
Re: [DISCUSS] KIP-106 - Change Default unclean.leader.election.enabled
from True to False
Posted by Joel Koshy <jj...@gmail.com>.
+1
On Tue, Jan 3, 2017 at 10:54 AM, Ben Stopford <be...@confluent.io> wrote:
> Hi All
>
> Please find the below KIP which proposes changing the setting
> unclean.leader.election.enabled from true to false. The motivation for
> this change is that it catches out new Kafka users who don’t realise the
> default favours availability over data loss.
>
> This would mean clusters wishing to continue with unclean leader election
> enabled would need to add the appropriate configuration on upgrade.
>
> Please let me know if you foresee any issue with this change, agree or
> don’t agree.
>
> https://cwiki.apache.org/confluence/display/KAFKA/%5BWIP%5D+
> KIP-106+-+Change+Default+unclean.leader.election.
> enabled+from+True+to+False <https://cwiki.apache.org/conf
> luence/display/KAFKA/[WIP]+KIP-106+-+Change+Default+uncle
> an.leader.election.enabled+from+True+to+False
> <https://cwiki.apache.org/confluence/display/KAFKA/%5BWIP%5D+KIP-106+-+Change+Default+unclean.leader.election.enabled+from+True+to+False>
> >
>
> Thanks
>
> B
>
> Ben Stopford
> Confluent, http://www.confluent.io <http://www.confluent.io/>
>
>
>
>