You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Prateek Agarwal <pr...@gmail.com> on 2020/11/25 17:36:08 UTC
[DISCUSS] KIP-688: Support dynamic update of delete.topic.enable config
Hi,
I've created KIP-688 to enable support for dynamic update of
delete.topic.enable config.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-688%3A+Support+dynamic+update+of+delete.topic.enable+config
Please take a look and let me know if you have any feedback.
Thanks
--
Prateek Agarwal
Re: [DISCUSS] KIP-688: Support dynamic update of delete.topic.enable config
Posted by Gwen Shapira <gw...@confluent.io>.
Harsha,
I understand the concern, but the solution of using this flag is a bit weird.
First, I can cause similar damage by setting retention.ms=1 on any
topic. We also have APIs to delete data from topics directly. All
without a "gating" flag. Because as a system we rely on access
controls to make sure unauthorized users can't cause excessive damage.
Second, I can think of no other system with a global flag for
enabling/disabling deletes - no OS does that (deleting files from
Linux is also a potential disaster) and no database. The fact that
Kafka is rather unique in this makes the flag and its default seem
like a trap for new users and an endless source of confusion.
Keeping a feature flag around for deletion seems to me to have more
downsides than upsides. If we want to add soft deletes to Kafka, we
can discuss that proposal - at least it is something users will be
familiar with.
On Wed, Nov 25, 2020 at 11:54 AM Harsha Ch <ha...@gmail.com> wrote:
>
> Hi Gwen,
> Agree with you ACLs would be a better option. But not
> all installations will be able to have secure setup and authorizer.
> In a hosted service where they may not be ACLs or security setup this flag
> helps prevent accidental topic deletion.
> This config acted as a good gatekeeper to prevent accidents in deletion as
> there is no confirmation or there is time delay in execution of the
> deletion of a topic or a soft delete etc...
> It's pretty much a catastrophic event in case of accidental deletion. We
> also need to consider that not all installations will be secure and we need
> to
> provide a simpler way to gate keep certain functionalities such as delete
> topic.
>
> Thanks,
> Harsha
>
>
> On Wed, Nov 25, 2020 at 10:25 AM Gwen Shapira <gw...@confluent.io> wrote:
>
> > Thanks for bringing this up!
> >
> > I disagree that " there is a valid use-case to keep the flag
> > "delete.topic.enable" as "false" in server.properties configs, so that
> > someone doesn't accidentally delete a topic,".
> > The flag was added and set to "false" back in the days when topic
> > deletion was not a stable feature, and enabling it meant a risk of
> > encountering serious bugs. Things changed in the last 4-5 years:
> > - Topic deletion is now a stable and reliable feature. We have it
> > enabled by default in Confluent Cloud and over a few years and
> > hundreds of clusters, there was maybe one issue related to deletion of
> > large number of partitions, and that was fixed.
> > - We have authorization and ACLs, so the permissions to delete topics
> > can be granted on an individual basis. This is the correct way to
> > handle concerns about someone accidentally deleting data valuable
> > data. Note that ACLs are dynamic.
> >
> > IMO, the best solution is to use the upcoming major release (3.0) to
> > enable topic deletion by default and remove the flag entirely.
> >
> > Gwen
> >
> > On Wed, Nov 25, 2020 at 9:36 AM Prateek Agarwal <pr...@gmail.com>
> > wrote:
> > >
> > > Hi,
> > >
> > > I've created KIP-688 to enable support for dynamic update of
> > > delete.topic.enable config.
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-688%3A+Support+dynamic+update+of+delete.topic.enable+config
> > >
> > > Please take a look and let me know if you have any feedback.
> > >
> > > Thanks
> > > --
> > > Prateek Agarwal
> >
> >
> >
> > --
> > Gwen Shapira
> > Engineering Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
--
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog
Re: [DISCUSS] KIP-688: Support dynamic update of delete.topic.enable config
Posted by Harsha Ch <ha...@gmail.com>.
Hi Gwen,
Agree with you ACLs would be a better option. But not
all installations will be able to have secure setup and authorizer.
In a hosted service where they may not be ACLs or security setup this flag
helps prevent accidental topic deletion.
This config acted as a good gatekeeper to prevent accidents in deletion as
there is no confirmation or there is time delay in execution of the
deletion of a topic or a soft delete etc...
It's pretty much a catastrophic event in case of accidental deletion. We
also need to consider that not all installations will be secure and we need
to
provide a simpler way to gate keep certain functionalities such as delete
topic.
Thanks,
Harsha
On Wed, Nov 25, 2020 at 10:25 AM Gwen Shapira <gw...@confluent.io> wrote:
> Thanks for bringing this up!
>
> I disagree that " there is a valid use-case to keep the flag
> "delete.topic.enable" as "false" in server.properties configs, so that
> someone doesn't accidentally delete a topic,".
> The flag was added and set to "false" back in the days when topic
> deletion was not a stable feature, and enabling it meant a risk of
> encountering serious bugs. Things changed in the last 4-5 years:
> - Topic deletion is now a stable and reliable feature. We have it
> enabled by default in Confluent Cloud and over a few years and
> hundreds of clusters, there was maybe one issue related to deletion of
> large number of partitions, and that was fixed.
> - We have authorization and ACLs, so the permissions to delete topics
> can be granted on an individual basis. This is the correct way to
> handle concerns about someone accidentally deleting data valuable
> data. Note that ACLs are dynamic.
>
> IMO, the best solution is to use the upcoming major release (3.0) to
> enable topic deletion by default and remove the flag entirely.
>
> Gwen
>
> On Wed, Nov 25, 2020 at 9:36 AM Prateek Agarwal <pr...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > I've created KIP-688 to enable support for dynamic update of
> > delete.topic.enable config.
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-688%3A+Support+dynamic+update+of+delete.topic.enable+config
> >
> > Please take a look and let me know if you have any feedback.
> >
> > Thanks
> > --
> > Prateek Agarwal
>
>
>
> --
> Gwen Shapira
> Engineering Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
Re: [DISCUSS] KIP-688: Support dynamic update of delete.topic.enable config
Posted by Gwen Shapira <gw...@confluent.io>.
Thanks for bringing this up!
I disagree that " there is a valid use-case to keep the flag
"delete.topic.enable" as "false" in server.properties configs, so that
someone doesn't accidentally delete a topic,".
The flag was added and set to "false" back in the days when topic
deletion was not a stable feature, and enabling it meant a risk of
encountering serious bugs. Things changed in the last 4-5 years:
- Topic deletion is now a stable and reliable feature. We have it
enabled by default in Confluent Cloud and over a few years and
hundreds of clusters, there was maybe one issue related to deletion of
large number of partitions, and that was fixed.
- We have authorization and ACLs, so the permissions to delete topics
can be granted on an individual basis. This is the correct way to
handle concerns about someone accidentally deleting data valuable
data. Note that ACLs are dynamic.
IMO, the best solution is to use the upcoming major release (3.0) to
enable topic deletion by default and remove the flag entirely.
Gwen
On Wed, Nov 25, 2020 at 9:36 AM Prateek Agarwal <pr...@gmail.com> wrote:
>
> Hi,
>
> I've created KIP-688 to enable support for dynamic update of
> delete.topic.enable config.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-688%3A+Support+dynamic+update+of+delete.topic.enable+config
>
> Please take a look and let me know if you have any feedback.
>
> Thanks
> --
> Prateek Agarwal
--
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog