You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Gaurav Narula <ka...@gnarula.com> on 2025/04/23 07:39:09 UTC

Re: [DISCUSS] KIP-1119: Add support for SSL hot reload

Hi Moncef,

Have you tried experimenting the behaviour of JDK's WatchService with bind mounts?

Reading [0], it appears that inotify on linux doesn't emit events for bind mounts. This may pose a limitation in Kubernetes as that's how config maps are mounted.

Another scenario to consider would be symlinks and how WatchService handles events on them.

Regards,
Gaurav

[0]: https://blog.arkey.fr/2019/09/13/watchservice-and-bind-mount/


> On 21 Mar 2025, at 04:40, Moncef Abboud <mo...@gmail.com> wrote:
> 
> Hi Mickael,
> 
> Thank you for your feedback.
> 
>> - Can you provide the description and default value for the new
> configuration?
> 
> Done. I amended the KIP.
> 
>> The KIP explicitly mentions brokers, consumer and producers, I assume it
> also covers admin clients (and controllers)?
> 
> Correct. I believe both the controllers and admin clients would benefit
> from the change since they use the SslFactory.
> 
>> The KIP states "To prevent redundant reconfigurations, a quiet period is
> enforced across watched files". How long is this period? Is it configurable?
> 
> The period is 30 seconds. My reasoning is that this should be long enough
> to allow an agent (such as a k8s Vault sidecar) to complete a few round
> trips with a remote server, but not too long. Given that each SslFactory
> (broker listener, client) has its own SSL properties, I thought it best to
> have this quiet period be consistent across all factories since there will
> only be one watcher thread. I chose not to make it configurable (one less
> thing to worry about), but I’m happy to change that.
> 
>> I'm unsure about the configuration name. Have you considered alternative
> like "ssl.auto.reload" or "ssl.stores.auto.reload"?
> Agreed. I changed it to "ssl.auto.reload".
> 
> Let me know what you think.
> 
> Thanks again,
> Moncef
> 
> On Wed, Mar 12, 2025 at 11:08 AM Mickael Maison <mi...@gmail.com>
> wrote:
> 
>> Hi,
>> 
>> Thanks for the KIP, it seems a useful improvement. I guess this would
>> supersede KIP-687.
>> 
>> The KIP is a bit light on details.
>> - Can you provide the description and default value for the new
>> configuration?
>> - The KIP explicitly mentions brokers, consumer and producers, I
>> assume it also covers admin clients (and controllers)?
>> - The KIP states "To prevent redundant reconfigurations, a quiet
>> period is enforced across watched files". How long is this period? Is
>> it configurable?
>> - I'm unsure about the configuration name. Have you considered
>> alternative like "ssl.auto.reload" or "ssl.stores.auto.reload"?
>> 
>> Thanks,
>> Mickael
>> 
>> On Thu, Dec 5, 2024 at 6:28 PM Moncef Abboud <mo...@gmail.com>
>> wrote:
>>> 
>>> Hi Gaurav,
>>> 
>>> Thank you for your reply.
>>> 
>>> There is some overlap between the two. However, while KIP-687 focuses
>>> primarily on brokers, this KIP targets consumers and producers, with
>>> additional benefits for brokers.
>>> 
>>> Best,
>>> Moncef
>>> 
>>> On Thu, Dec 5, 2024, 10:38 AM Gaurav Narula <ka...@gnarula.com> wrote:
>>> 
>>>> Hi Moncef,
>>>> 
>>>> Thank you for the KIP. It seems very similar in spirit to KIP-687 (
>>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-687%3A+Automatic+Reloading+of+Security+Store
>> )
>>>> which seems like it was approved but never fully implemented. Can you
>>>> please confirm if it is the case indeed?
>>>> 
>>>> Regards,
>>>> Gaurav
>>>> 
>>>>> On 2 Dec 2024, at 23:12, Moncef Abboud <mo...@gmail.com>
>>>> wrote:
>>>>> 
>>>>> Hi all,
>>>>> 
>>>>> I hope your week is off to a great start.
>>>>> 
>>>>> I created a KIP to add support for SSL hot reloading.
>>>>> https://cwiki.apache.org/confluence/x/eIrREw
>>>>> 
>>>>> Thank you for your feedback!
>>>>> 
>>>>> 
>>>>> Moncef
>>>> 
>>>> 
>> 
> 
> 
> -- 
> Moncef  ABBOUD


Re: [DISCUSS] KIP-1119: Add support for SSL hot reload

Posted by Moncef Abboud <mo...@gmail.com>.
Gaurav, Colin,

Thank you for your replies.

> It appears that inotify on Linux doesn't emit events for bind mounts.

This is indeed a valid concern and it seems that this a well-known issue..
As Colin suggested, we can work around this by manually polling or by
leveraging a library that uses polling behind the scenes, such as Apache
Commons IO Monitor [0].

Regarding the symlinks, we can add a check to resolve symlinks to their
actual paths (java.nio.file.Files.readSymbolicLink).
What do you think?

Thanks,
Moncef

[0]
https://github.com/apache/commons-io/blob/6ddc0834e63d0924ae1bca939058ac749214f2fb/src/main/java/org/apache/commons/io/monitor/FileAlterationObserver.java#L531



On Tue, Apr 29, 2025 at 3:09 PM Colin McCabe <cm...@apache.org> wrote:

> Gaurav brings up a good point. However, we can work around that bug by
> doing polling every 5 or 10 minutes in addition to using WatchService.
>
> best,
> Colin
>
> On Wed, Apr 23, 2025, at 00:39, Gaurav Narula wrote:
> > Hi Moncef,
> >
> > Have you tried experimenting the behaviour of JDK's WatchService with
> > bind mounts?
> >
> > Reading [0], it appears that inotify on linux doesn't emit events for
> > bind mounts. This may pose a limitation in Kubernetes as that's how
> > config maps are mounted.
> >
> > Another scenario to consider would be symlinks and how WatchService
> > handles events on them.
> >
> > Regards,
> > Gaurav
> >
> > [0]: https://blog.arkey.fr/2019/09/13/watchservice-and-bind-mount/
> >
> >
> >> On 21 Mar 2025, at 04:40, Moncef Abboud <mo...@gmail.com>
> wrote:
> >>
> >> Hi Mickael,
> >>
> >> Thank you for your feedback.
> >>
> >>> - Can you provide the description and default value for the new
> >> configuration?
> >>
> >> Done. I amended the KIP.
> >>
> >>> The KIP explicitly mentions brokers, consumer and producers, I assume
> it
> >> also covers admin clients (and controllers)?
> >>
> >> Correct. I believe both the controllers and admin clients would benefit
> >> from the change since they use the SslFactory.
> >>
> >>> The KIP states "To prevent redundant reconfigurations, a quiet period
> is
> >> enforced across watched files". How long is this period? Is it
> configurable?
> >>
> >> The period is 30 seconds. My reasoning is that this should be long
> enough
> >> to allow an agent (such as a k8s Vault sidecar) to complete a few round
> >> trips with a remote server, but not too long. Given that each SslFactory
> >> (broker listener, client) has its own SSL properties, I thought it best
> to
> >> have this quiet period be consistent across all factories since there
> will
> >> only be one watcher thread. I chose not to make it configurable (one
> less
> >> thing to worry about), but I’m happy to change that.
> >>
> >>> I'm unsure about the configuration name. Have you considered
> alternative
> >> like "ssl.auto.reload" or "ssl.stores.auto.reload"?
> >> Agreed. I changed it to "ssl.auto.reload".
> >>
> >> Let me know what you think.
> >>
> >> Thanks again,
> >> Moncef
> >>
> >> On Wed, Mar 12, 2025 at 11:08 AM Mickael Maison <
> mickael.maison@gmail.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Thanks for the KIP, it seems a useful improvement. I guess this would
> >>> supersede KIP-687.
> >>>
> >>> The KIP is a bit light on details.
> >>> - Can you provide the description and default value for the new
> >>> configuration?
> >>> - The KIP explicitly mentions brokers, consumer and producers, I
> >>> assume it also covers admin clients (and controllers)?
> >>> - The KIP states "To prevent redundant reconfigurations, a quiet
> >>> period is enforced across watched files". How long is this period? Is
> >>> it configurable?
> >>> - I'm unsure about the configuration name. Have you considered
> >>> alternative like "ssl.auto.reload" or "ssl.stores.auto.reload"?
> >>>
> >>> Thanks,
> >>> Mickael
> >>>
> >>> On Thu, Dec 5, 2024 at 6:28 PM Moncef Abboud <
> moncef.abboud95@gmail.com>
> >>> wrote:
> >>>>
> >>>> Hi Gaurav,
> >>>>
> >>>> Thank you for your reply.
> >>>>
> >>>> There is some overlap between the two. However, while KIP-687 focuses
> >>>> primarily on brokers, this KIP targets consumers and producers, with
> >>>> additional benefits for brokers.
> >>>>
> >>>> Best,
> >>>> Moncef
> >>>>
> >>>> On Thu, Dec 5, 2024, 10:38 AM Gaurav Narula <ka...@gnarula.com>
> wrote:
> >>>>
> >>>>> Hi Moncef,
> >>>>>
> >>>>> Thank you for the KIP. It seems very similar in spirit to KIP-687 (
> >>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-687%3A+Automatic+Reloading+of+Security+Store
> >>> )
> >>>>> which seems like it was approved but never fully implemented. Can you
> >>>>> please confirm if it is the case indeed?
> >>>>>
> >>>>> Regards,
> >>>>> Gaurav
> >>>>>
> >>>>>> On 2 Dec 2024, at 23:12, Moncef Abboud <mo...@gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> I hope your week is off to a great start.
> >>>>>>
> >>>>>> I created a KIP to add support for SSL hot reloading.
> >>>>>> https://cwiki.apache.org/confluence/x/eIrREw
> >>>>>>
> >>>>>> Thank you for your feedback!
> >>>>>>
> >>>>>>
> >>>>>> Moncef
> >>>>>
> >>>>>
> >>>
> >>
> >>
> >> --
> >> Moncef  ABBOUD
>


-- 
Moncef  ABBOUD

Re: [DISCUSS] KIP-1119: Add support for SSL hot reload

Posted by Colin McCabe <cm...@apache.org>.
Gaurav brings up a good point. However, we can work around that bug by doing polling every 5 or 10 minutes in addition to using WatchService.

best,
Colin

On Wed, Apr 23, 2025, at 00:39, Gaurav Narula wrote:
> Hi Moncef,
>
> Have you tried experimenting the behaviour of JDK's WatchService with 
> bind mounts?
>
> Reading [0], it appears that inotify on linux doesn't emit events for 
> bind mounts. This may pose a limitation in Kubernetes as that's how 
> config maps are mounted.
>
> Another scenario to consider would be symlinks and how WatchService 
> handles events on them.
>
> Regards,
> Gaurav
>
> [0]: https://blog.arkey.fr/2019/09/13/watchservice-and-bind-mount/
>
>
>> On 21 Mar 2025, at 04:40, Moncef Abboud <mo...@gmail.com> wrote:
>> 
>> Hi Mickael,
>> 
>> Thank you for your feedback.
>> 
>>> - Can you provide the description and default value for the new
>> configuration?
>> 
>> Done. I amended the KIP.
>> 
>>> The KIP explicitly mentions brokers, consumer and producers, I assume it
>> also covers admin clients (and controllers)?
>> 
>> Correct. I believe both the controllers and admin clients would benefit
>> from the change since they use the SslFactory.
>> 
>>> The KIP states "To prevent redundant reconfigurations, a quiet period is
>> enforced across watched files". How long is this period? Is it configurable?
>> 
>> The period is 30 seconds. My reasoning is that this should be long enough
>> to allow an agent (such as a k8s Vault sidecar) to complete a few round
>> trips with a remote server, but not too long. Given that each SslFactory
>> (broker listener, client) has its own SSL properties, I thought it best to
>> have this quiet period be consistent across all factories since there will
>> only be one watcher thread. I chose not to make it configurable (one less
>> thing to worry about), but I’m happy to change that.
>> 
>>> I'm unsure about the configuration name. Have you considered alternative
>> like "ssl.auto.reload" or "ssl.stores.auto.reload"?
>> Agreed. I changed it to "ssl.auto.reload".
>> 
>> Let me know what you think.
>> 
>> Thanks again,
>> Moncef
>> 
>> On Wed, Mar 12, 2025 at 11:08 AM Mickael Maison <mi...@gmail.com>
>> wrote:
>> 
>>> Hi,
>>> 
>>> Thanks for the KIP, it seems a useful improvement. I guess this would
>>> supersede KIP-687.
>>> 
>>> The KIP is a bit light on details.
>>> - Can you provide the description and default value for the new
>>> configuration?
>>> - The KIP explicitly mentions brokers, consumer and producers, I
>>> assume it also covers admin clients (and controllers)?
>>> - The KIP states "To prevent redundant reconfigurations, a quiet
>>> period is enforced across watched files". How long is this period? Is
>>> it configurable?
>>> - I'm unsure about the configuration name. Have you considered
>>> alternative like "ssl.auto.reload" or "ssl.stores.auto.reload"?
>>> 
>>> Thanks,
>>> Mickael
>>> 
>>> On Thu, Dec 5, 2024 at 6:28 PM Moncef Abboud <mo...@gmail.com>
>>> wrote:
>>>> 
>>>> Hi Gaurav,
>>>> 
>>>> Thank you for your reply.
>>>> 
>>>> There is some overlap between the two. However, while KIP-687 focuses
>>>> primarily on brokers, this KIP targets consumers and producers, with
>>>> additional benefits for brokers.
>>>> 
>>>> Best,
>>>> Moncef
>>>> 
>>>> On Thu, Dec 5, 2024, 10:38 AM Gaurav Narula <ka...@gnarula.com> wrote:
>>>> 
>>>>> Hi Moncef,
>>>>> 
>>>>> Thank you for the KIP. It seems very similar in spirit to KIP-687 (
>>>>> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-687%3A+Automatic+Reloading+of+Security+Store
>>> )
>>>>> which seems like it was approved but never fully implemented. Can you
>>>>> please confirm if it is the case indeed?
>>>>> 
>>>>> Regards,
>>>>> Gaurav
>>>>> 
>>>>>> On 2 Dec 2024, at 23:12, Moncef Abboud <mo...@gmail.com>
>>>>> wrote:
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> I hope your week is off to a great start.
>>>>>> 
>>>>>> I created a KIP to add support for SSL hot reloading.
>>>>>> https://cwiki.apache.org/confluence/x/eIrREw
>>>>>> 
>>>>>> Thank you for your feedback!
>>>>>> 
>>>>>> 
>>>>>> Moncef
>>>>> 
>>>>> 
>>> 
>> 
>> 
>> -- 
>> Moncef  ABBOUD