You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Gaurav Narula <ka...@gnarula.com> on 2025/04/23 07:39:09 UTC
Re: [DISCUSS] KIP-1119: Add support for SSL hot reload
Hi Moncef,
Have you tried experimenting the behaviour of JDK's WatchService with bind mounts?
Reading [0], it appears that inotify on linux doesn't emit events for bind mounts. This may pose a limitation in Kubernetes as that's how config maps are mounted.
Another scenario to consider would be symlinks and how WatchService handles events on them.
Regards,
Gaurav
[0]: https://blog.arkey.fr/2019/09/13/watchservice-and-bind-mount/
> On 21 Mar 2025, at 04:40, Moncef Abboud <mo...@gmail.com> wrote:
>
> Hi Mickael,
>
> Thank you for your feedback.
>
>> - Can you provide the description and default value for the new
> configuration?
>
> Done. I amended the KIP.
>
>> The KIP explicitly mentions brokers, consumer and producers, I assume it
> also covers admin clients (and controllers)?
>
> Correct. I believe both the controllers and admin clients would benefit
> from the change since they use the SslFactory.
>
>> The KIP states "To prevent redundant reconfigurations, a quiet period is
> enforced across watched files". How long is this period? Is it configurable?
>
> The period is 30 seconds. My reasoning is that this should be long enough
> to allow an agent (such as a k8s Vault sidecar) to complete a few round
> trips with a remote server, but not too long. Given that each SslFactory
> (broker listener, client) has its own SSL properties, I thought it best to
> have this quiet period be consistent across all factories since there will
> only be one watcher thread. I chose not to make it configurable (one less
> thing to worry about), but I’m happy to change that.
>
>> I'm unsure about the configuration name. Have you considered alternative
> like "ssl.auto.reload" or "ssl.stores.auto.reload"?
> Agreed. I changed it to "ssl.auto.reload".
>
> Let me know what you think.
>
> Thanks again,
> Moncef
>
> On Wed, Mar 12, 2025 at 11:08 AM Mickael Maison <mi...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Thanks for the KIP, it seems a useful improvement. I guess this would
>> supersede KIP-687.
>>
>> The KIP is a bit light on details.
>> - Can you provide the description and default value for the new
>> configuration?
>> - The KIP explicitly mentions brokers, consumer and producers, I
>> assume it also covers admin clients (and controllers)?
>> - The KIP states "To prevent redundant reconfigurations, a quiet
>> period is enforced across watched files". How long is this period? Is
>> it configurable?
>> - I'm unsure about the configuration name. Have you considered
>> alternative like "ssl.auto.reload" or "ssl.stores.auto.reload"?
>>
>> Thanks,
>> Mickael
>>
>> On Thu, Dec 5, 2024 at 6:28 PM Moncef Abboud <mo...@gmail.com>
>> wrote:
>>>
>>> Hi Gaurav,
>>>
>>> Thank you for your reply.
>>>
>>> There is some overlap between the two. However, while KIP-687 focuses
>>> primarily on brokers, this KIP targets consumers and producers, with
>>> additional benefits for brokers.
>>>
>>> Best,
>>> Moncef
>>>
>>> On Thu, Dec 5, 2024, 10:38 AM Gaurav Narula <ka...@gnarula.com> wrote:
>>>
>>>> Hi Moncef,
>>>>
>>>> Thank you for the KIP. It seems very similar in spirit to KIP-687 (
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-687%3A+Automatic+Reloading+of+Security+Store
>> )
>>>> which seems like it was approved but never fully implemented. Can you
>>>> please confirm if it is the case indeed?
>>>>
>>>> Regards,
>>>> Gaurav
>>>>
>>>>> On 2 Dec 2024, at 23:12, Moncef Abboud <mo...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I hope your week is off to a great start.
>>>>>
>>>>> I created a KIP to add support for SSL hot reloading.
>>>>> https://cwiki.apache.org/confluence/x/eIrREw
>>>>>
>>>>> Thank you for your feedback!
>>>>>
>>>>>
>>>>> Moncef
>>>>
>>>>
>>
>
>
> --
> Moncef ABBOUD
Re: [DISCUSS] KIP-1119: Add support for SSL hot reload
Posted by Moncef Abboud <mo...@gmail.com>.
Gaurav, Colin,
Thank you for your replies.
> It appears that inotify on Linux doesn't emit events for bind mounts.
This is indeed a valid concern and it seems that this a well-known issue..
As Colin suggested, we can work around this by manually polling or by
leveraging a library that uses polling behind the scenes, such as Apache
Commons IO Monitor [0].
Regarding the symlinks, we can add a check to resolve symlinks to their
actual paths (java.nio.file.Files.readSymbolicLink).
What do you think?
Thanks,
Moncef
[0]
https://github.com/apache/commons-io/blob/6ddc0834e63d0924ae1bca939058ac749214f2fb/src/main/java/org/apache/commons/io/monitor/FileAlterationObserver.java#L531
On Tue, Apr 29, 2025 at 3:09 PM Colin McCabe <cm...@apache.org> wrote:
> Gaurav brings up a good point. However, we can work around that bug by
> doing polling every 5 or 10 minutes in addition to using WatchService.
>
> best,
> Colin
>
> On Wed, Apr 23, 2025, at 00:39, Gaurav Narula wrote:
> > Hi Moncef,
> >
> > Have you tried experimenting the behaviour of JDK's WatchService with
> > bind mounts?
> >
> > Reading [0], it appears that inotify on linux doesn't emit events for
> > bind mounts. This may pose a limitation in Kubernetes as that's how
> > config maps are mounted.
> >
> > Another scenario to consider would be symlinks and how WatchService
> > handles events on them.
> >
> > Regards,
> > Gaurav
> >
> > [0]: https://blog.arkey.fr/2019/09/13/watchservice-and-bind-mount/
> >
> >
> >> On 21 Mar 2025, at 04:40, Moncef Abboud <mo...@gmail.com>
> wrote:
> >>
> >> Hi Mickael,
> >>
> >> Thank you for your feedback.
> >>
> >>> - Can you provide the description and default value for the new
> >> configuration?
> >>
> >> Done. I amended the KIP.
> >>
> >>> The KIP explicitly mentions brokers, consumer and producers, I assume
> it
> >> also covers admin clients (and controllers)?
> >>
> >> Correct. I believe both the controllers and admin clients would benefit
> >> from the change since they use the SslFactory.
> >>
> >>> The KIP states "To prevent redundant reconfigurations, a quiet period
> is
> >> enforced across watched files". How long is this period? Is it
> configurable?
> >>
> >> The period is 30 seconds. My reasoning is that this should be long
> enough
> >> to allow an agent (such as a k8s Vault sidecar) to complete a few round
> >> trips with a remote server, but not too long. Given that each SslFactory
> >> (broker listener, client) has its own SSL properties, I thought it best
> to
> >> have this quiet period be consistent across all factories since there
> will
> >> only be one watcher thread. I chose not to make it configurable (one
> less
> >> thing to worry about), but I’m happy to change that.
> >>
> >>> I'm unsure about the configuration name. Have you considered
> alternative
> >> like "ssl.auto.reload" or "ssl.stores.auto.reload"?
> >> Agreed. I changed it to "ssl.auto.reload".
> >>
> >> Let me know what you think.
> >>
> >> Thanks again,
> >> Moncef
> >>
> >> On Wed, Mar 12, 2025 at 11:08 AM Mickael Maison <
> mickael.maison@gmail.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Thanks for the KIP, it seems a useful improvement. I guess this would
> >>> supersede KIP-687.
> >>>
> >>> The KIP is a bit light on details.
> >>> - Can you provide the description and default value for the new
> >>> configuration?
> >>> - The KIP explicitly mentions brokers, consumer and producers, I
> >>> assume it also covers admin clients (and controllers)?
> >>> - The KIP states "To prevent redundant reconfigurations, a quiet
> >>> period is enforced across watched files". How long is this period? Is
> >>> it configurable?
> >>> - I'm unsure about the configuration name. Have you considered
> >>> alternative like "ssl.auto.reload" or "ssl.stores.auto.reload"?
> >>>
> >>> Thanks,
> >>> Mickael
> >>>
> >>> On Thu, Dec 5, 2024 at 6:28 PM Moncef Abboud <
> moncef.abboud95@gmail.com>
> >>> wrote:
> >>>>
> >>>> Hi Gaurav,
> >>>>
> >>>> Thank you for your reply.
> >>>>
> >>>> There is some overlap between the two. However, while KIP-687 focuses
> >>>> primarily on brokers, this KIP targets consumers and producers, with
> >>>> additional benefits for brokers.
> >>>>
> >>>> Best,
> >>>> Moncef
> >>>>
> >>>> On Thu, Dec 5, 2024, 10:38 AM Gaurav Narula <ka...@gnarula.com>
> wrote:
> >>>>
> >>>>> Hi Moncef,
> >>>>>
> >>>>> Thank you for the KIP. It seems very similar in spirit to KIP-687 (
> >>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-687%3A+Automatic+Reloading+of+Security+Store
> >>> )
> >>>>> which seems like it was approved but never fully implemented. Can you
> >>>>> please confirm if it is the case indeed?
> >>>>>
> >>>>> Regards,
> >>>>> Gaurav
> >>>>>
> >>>>>> On 2 Dec 2024, at 23:12, Moncef Abboud <mo...@gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> I hope your week is off to a great start.
> >>>>>>
> >>>>>> I created a KIP to add support for SSL hot reloading.
> >>>>>> https://cwiki.apache.org/confluence/x/eIrREw
> >>>>>>
> >>>>>> Thank you for your feedback!
> >>>>>>
> >>>>>>
> >>>>>> Moncef
> >>>>>
> >>>>>
> >>>
> >>
> >>
> >> --
> >> Moncef ABBOUD
>
--
Moncef ABBOUD
Re: [DISCUSS] KIP-1119: Add support for SSL hot reload
Posted by Colin McCabe <cm...@apache.org>.
Gaurav brings up a good point. However, we can work around that bug by doing polling every 5 or 10 minutes in addition to using WatchService.
best,
Colin
On Wed, Apr 23, 2025, at 00:39, Gaurav Narula wrote:
> Hi Moncef,
>
> Have you tried experimenting the behaviour of JDK's WatchService with
> bind mounts?
>
> Reading [0], it appears that inotify on linux doesn't emit events for
> bind mounts. This may pose a limitation in Kubernetes as that's how
> config maps are mounted.
>
> Another scenario to consider would be symlinks and how WatchService
> handles events on them.
>
> Regards,
> Gaurav
>
> [0]: https://blog.arkey.fr/2019/09/13/watchservice-and-bind-mount/
>
>
>> On 21 Mar 2025, at 04:40, Moncef Abboud <mo...@gmail.com> wrote:
>>
>> Hi Mickael,
>>
>> Thank you for your feedback.
>>
>>> - Can you provide the description and default value for the new
>> configuration?
>>
>> Done. I amended the KIP.
>>
>>> The KIP explicitly mentions brokers, consumer and producers, I assume it
>> also covers admin clients (and controllers)?
>>
>> Correct. I believe both the controllers and admin clients would benefit
>> from the change since they use the SslFactory.
>>
>>> The KIP states "To prevent redundant reconfigurations, a quiet period is
>> enforced across watched files". How long is this period? Is it configurable?
>>
>> The period is 30 seconds. My reasoning is that this should be long enough
>> to allow an agent (such as a k8s Vault sidecar) to complete a few round
>> trips with a remote server, but not too long. Given that each SslFactory
>> (broker listener, client) has its own SSL properties, I thought it best to
>> have this quiet period be consistent across all factories since there will
>> only be one watcher thread. I chose not to make it configurable (one less
>> thing to worry about), but I’m happy to change that.
>>
>>> I'm unsure about the configuration name. Have you considered alternative
>> like "ssl.auto.reload" or "ssl.stores.auto.reload"?
>> Agreed. I changed it to "ssl.auto.reload".
>>
>> Let me know what you think.
>>
>> Thanks again,
>> Moncef
>>
>> On Wed, Mar 12, 2025 at 11:08 AM Mickael Maison <mi...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Thanks for the KIP, it seems a useful improvement. I guess this would
>>> supersede KIP-687.
>>>
>>> The KIP is a bit light on details.
>>> - Can you provide the description and default value for the new
>>> configuration?
>>> - The KIP explicitly mentions brokers, consumer and producers, I
>>> assume it also covers admin clients (and controllers)?
>>> - The KIP states "To prevent redundant reconfigurations, a quiet
>>> period is enforced across watched files". How long is this period? Is
>>> it configurable?
>>> - I'm unsure about the configuration name. Have you considered
>>> alternative like "ssl.auto.reload" or "ssl.stores.auto.reload"?
>>>
>>> Thanks,
>>> Mickael
>>>
>>> On Thu, Dec 5, 2024 at 6:28 PM Moncef Abboud <mo...@gmail.com>
>>> wrote:
>>>>
>>>> Hi Gaurav,
>>>>
>>>> Thank you for your reply.
>>>>
>>>> There is some overlap between the two. However, while KIP-687 focuses
>>>> primarily on brokers, this KIP targets consumers and producers, with
>>>> additional benefits for brokers.
>>>>
>>>> Best,
>>>> Moncef
>>>>
>>>> On Thu, Dec 5, 2024, 10:38 AM Gaurav Narula <ka...@gnarula.com> wrote:
>>>>
>>>>> Hi Moncef,
>>>>>
>>>>> Thank you for the KIP. It seems very similar in spirit to KIP-687 (
>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-687%3A+Automatic+Reloading+of+Security+Store
>>> )
>>>>> which seems like it was approved but never fully implemented. Can you
>>>>> please confirm if it is the case indeed?
>>>>>
>>>>> Regards,
>>>>> Gaurav
>>>>>
>>>>>> On 2 Dec 2024, at 23:12, Moncef Abboud <mo...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I hope your week is off to a great start.
>>>>>>
>>>>>> I created a KIP to add support for SSL hot reloading.
>>>>>> https://cwiki.apache.org/confluence/x/eIrREw
>>>>>>
>>>>>> Thank you for your feedback!
>>>>>>
>>>>>>
>>>>>> Moncef
>>>>>
>>>>>
>>>
>>
>>
>> --
>> Moncef ABBOUD