You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by 黃竣陽 <s7...@gmail.com> on 2025/10/06 12:28:46 UTC
Re: [VOTE] KIP-877: Mechanism for plugins and connectors to register metrics
Hello,
I haven’t seen any feedback, so I’ll go ahead and implement the improvement.
Best Regards,
Jiunn-Yang
> 黃竣陽 <s7...@gmail.com> 於 2025年9月29日 晚上11:26 寫道：
>
> Hello all,
>
> There is an improvement in KIP-877 with the following changes:
>
> 1. Removed KafkaPrincipalBuilder plugin: Its lifecycle is short,
> so adding metrics for it would be meaningless.
>
> 2. Added new extra tags for Login, and AuthenticateCallbackHandler
> plugins to avoid collisions. These plugins now define metrics under three different scenarios:
>
> a. Server component, ConnectionMode = Server → registers: "mechanism", "networkProcessor", "listener"
> b. Server component, ConnectionMode = Client → registers: "component", "node-id"
> c. Client component, ConnectionMode = Client → no extra tags
>
> 3. Added new extra tags for SslEngineFactory
> plugins to avoid collisions. These plugins now define metrics under three different scenarios:
>
> a. Server component, ConnectionMode = Server → registers: "networkProcessor", "listener"
> b. Server component, ConnectionMode = Client → registers: "component", "node-id"
> c. Client component, ConnectionMode = Client → no extra tags
>
> <https://cwiki.apache.org/confluence/x/lY3GDQ>
>
> Best Regards,
> Jiunn-Yang
>
>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨2:47 寫道：
>>
>> Hi, Jiunn-Yang,
>>
>> Thanks for the explanation. The change sounds good to me then.
>>
>> Jun
>>
>> On Wed, Apr 23, 2025 at 4:57 AM 黃竣陽 <s7...@gmail.com> wrote:
>>
>>> Hello Jun,
>>>
>>> At this stage, KAFKA-18390 only involves modifications to internal data
>>> structures. Interface-level changes
>>> are currently out of scope. If any interface changes in the future, they
>>> will be tracked under a separate JIRA.
>>>
>>> Best Regards,
>>> Jiunn-Yang
>>>
>>>> Mickael Maison <mi...@gmail.com> 於 2025年4月23日 下午6:07 寫道：
>>>>
>>>> Hi Jun,
>>>>
>>>> KIP-877 is not in a release yet. So assuming nobody objects to the
>>>> change, we can do it before 4.1.0.
>>>> Then any further changes will require another KIP.
>>>>
>>>> Thanks,
>>>> Mickael
>>>>
>>>>
>>>> On Tue, Apr 22, 2025 at 8:05 PM Jun Rao <ju...@confluent.io.invalid>
>>> wrote:
>>>>>
>>>>> Hi, Jiunn-Yang,
>>>>>
>>>>> Thanks for the improvement.
>>>>>
>>>>> KAFKA-18390 changes a public interface. Do we have a KIP for that?
>>>>>
>>>>> Jun
>>>>>
>>>>> On Tue, Apr 22, 2025 at 3:39 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I haven’t seen any feedback, so I’ll go ahead and implement the
>>>>>> improvement
>>>>>> regarding PluginMetrics#metricName.
>>>>>>
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>>
>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年4月14日 晚上11:43 寫道：
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> There is an improvement in KIP-877.
>>>>>>>
>>>>>>> There will be an update to the PluginMetrics#metricName method: the
>>> type
>>>>>> of the tags parameter will be changed
>>>>>>> from Map to LinkedHashMap.
>>>>>>> This change is necessary because the order of metric tags is important
>>>>>>> 1. If the tag order is inconsistent, identical metrics may be treated
>>> as
>>>>>> distinct ones by the metrics backend
>>>>>>> 2. KAFKA-18390 is updating metric naming to use LinkedHashMap. For
>>>>>> consistency, we should follow the same approach here.
>>>>>>>
>>>>>>> <https://cwiki.apache.org/confluence/x/lY3GDQ>
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Jiunn-Yang
>>>>>>>
>>>>>>>> Luke Chen <sh...@gmail.com> 於 2025年3月12日 上午10:21 寫道：
>>>>>>>>
>>>>>>>> HI Mickael,
>>>>>>>>
>>>>>>>> Thanks for the update.
>>>>>>>> The extra tags addition makes sense to me.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>> Luke
>>>>>>>>
>>>>>>>> On Tue, Mar 11, 2025 at 8:31 PM Mickael Maison <
>>>>>> mickael.maison@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I've not seen any concerns or feedback.
>>>>>>>>> I'll wait a few more days, then we'll carry on the implementation of
>>>>>>>>> KIP-877 with the updated proposal (extra tags for some server side
>>>>>>>>> plugin).
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Mickael
>>>>>>>>>
>>>>>>>>> On Tue, Mar 4, 2025 at 6:57 PM Mickael Maison <
>>>>>> mickael.maison@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I found an issue while implementing KIP-877 for server plugins.
>>>>>>>>>>
>>>>>>>>>> Several server plugins can be instantiated multiple times. This can
>>>>>>>>> happen:
>>>>>>>>>> - when a server is running in combined mode (for example
>>> Authorizer is
>>>>>>>>>> instantiated by both brokers and controllers)
>>>>>>>>>> - when using multiple listeners (for example KafkaPrincipalBuilder
>>> is
>>>>>>>>>> instantiated per processor per listener)
>>>>>>>>>> In the voted proposal, I did not include enough tags for some
>>> plugins
>>>>>>>>>> to uniquely identify each instance and avoid metric name
>>> collisions.
>>>>>>>>>>
>>>>>>>>>> For that reason, I updated the KIP to specify extra tags (in
>>> addition
>>>>>>>>>> of "config" and "class") for the following plugin types:
>>>>>>>>>> - KafkaPrincipalBuilder
>>>>>>>>>> - Authorizer
>>>>>>>>>> - ClientQuotaCallback
>>>>>>>>>> - ConfigProvider
>>>>>>>>>> - AuthenticateCallbackHandler
>>>>>>>>>> - Login
>>>>>>>>>> - SslEngineFactory
>>>>>>>>>>
>>>>>>>>>> You can find the exact tags in the Proposed Changes section of the
>>>>>> KIP:
>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=231116181#KIP877:Mechanismforpluginsandconnectorstoregistermetrics-ProposedChanges
>>>>>>>>>>
>>>>>>>>>> Let me know if you have questions or concerns.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Mickael
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 14, 2024 at 3:07 PM Mickael Maison <
>>>>>> mickael.maison@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> While implementing the KIP I noticed the Kafka Connect specifics
>>> had
>>>>>>>>>>> not been updated correctly based on the latest discussions. The
>>>>>>>>>>> consensus was that connector and tasks should use their context
>>> to
>>>>>>>>>>> retrieve the PluginMetrics instance if they want to register
>>> metrics.
>>>>>>>>>>> While it's a slightly different mechanism than for the other
>>> plugins
>>>>>>>>>>> (they need to implement Monitorable), this allows connector to
>>>>>> support
>>>>>>>>>>> this new feature and still work when deployed on an older Connect
>>>>>>>>>>> runtime.
>>>>>>>>>>>
>>>>>>>>>>> I have updated the KIP and I have 2 PRs ready for review:
>>>>>>>>>>> - Adding the APIs and support for producer and consumer plugins:
>>>>>>>>>>> https://github.com/apache/kafka/pull/17511
>>>>>>>>>>> - Adding support for Connect plugins:
>>>>>>>>> https://github.com/apache/kafka/pull/17804
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Mickael
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:21 PM Mickael Maison <
>>>>>>>>> mickael.maison@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm closing the vote. The vote has passed with
>>>>>>>>>>>> + 3 bindings votes from Chris, Tom and Luke, and +1 non-binding
>>> vote
>>>>>>>>>>>> from Hector.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Mickael
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 8, 2024 at 11:28 AM Luke Chen <sh...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Mickael,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>> +1 (binding) from me.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>> Luke
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Aug 2, 2024 at 4:57 AM Tom Bentley <tbentley@redhat.com
>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> +1 (binding).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks Mickael!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, 1 Aug 2024 at 05:12, Mickael Maison <
>>>>>>>>> mickael.maison@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Bumping this thread to get some more votes and/or feedback.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As I restarted the vote on June 10 after major changes, I'm
>>>>>>>>> only
>>>>>>>>>>>>>>> counting votes since then.
>>>>>>>>>>>>>>> So we have 1 binding (Chris) and 1 non-binding (Hector) votes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Mickael
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Jul 8, 2024 at 8:40 PM Hector Geraldino (BLOOMBERG/
>>>>>>>>> 919 3RD A)
>>>>>>>>>>>>>>> <hg...@bloomberg.net> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This will help eliminate some boilerplate code we have for
>>>>>>>>> our
>>>>>>>>>>>>>>> connectors.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +1 (non-binding)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> From: dev@kafka.apache.org At: 06/25/24 04:30:27 UTC-4:00To:
>>>>>>>>>>>>>>> dev@kafka.apache.org
>>>>>>>>>>>>>>>> Subject: Re: [VOTE] KIP-877: Mechanism for plugins and
>>>>>>>>> connectors to
>>>>>>>>>>>>>>> register metrics
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Bumping this thread.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Let me know if you have any feedback.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Mickael
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Jun 10, 2024 at 1:44 PM Chris Egerton <
>>>>>>>>> fearthecellos@gmail.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> +1 (binding), thanks Mickael!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Jun 10, 2024, 04:24 Mickael Maison <
>>>>>>>>> mickael.maison@gmail.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Following the feedback in the DISCUSS thread, I made
>>>>>>>>> significant
>>>>>>>>>>>>>>>>>> changes to the proposal. So I'd like to restart a vote
>>>>>>>>> for KIP-877:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism+for+plugi
>>>>>>>>>>>>>>>> ns+and+connectors+to+register+metrics
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Mickael
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Jan 25, 2024 at 2:59 AM Tom Bentley <
>>>>>>>>> tbentley@redhat.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Mickael,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> You'll have seen that I left some comments on the
>>>>>>>>> discussion
>>>>>>>>>>>>>>> thread, but
>>>>>>>>>>>>>>>>>>> they're minor enough that I'm happy to vote +1 here.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Tom
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Thu, 11 Jan 2024 at 06:14, Mickael Maison <
>>>>>>>>>>>>>>> mickael.maison@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Bumping this thread since I've not seen any feedback.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Mickael
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Dec 19, 2023 at 10:03 AM Mickael Maison
>>>>>>>>>>>>>>>>>>>> <mi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I'd like to start a vote on KIP-877:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism+for+plugi
>>>>>>>>>>>>>>>> ns+and+connectors+to+register+metrics
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Let me know if you have any feedback.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> Mickael
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>
>>>
>