You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Cyrus Vafadari <cy...@confluent.io> on 2019/05/30 06:06:12 UTC
[DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Hello Dev,
I'd like to start the discussion of KIP-475: New Metric to Measure Number
of Tasks on a Connector.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector
The proposal is pretty straightforward -- to add a new metric to Connect to
measure the number of tasks on a Connector. Currently, we support this on
Worker level, so this KIP just adds another metric to support this
per-connector.
There is also a PR:
https://github.com/apache/kafka/pull/6843
Thanks,
Cyrus
Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Posted by Cyrus Vafadari <cy...@confluent.io>.
Thanks, everyone, for the feedback.
It looks like there's support, so I'll open the voting thread.
Cyrus
On Tue, Jun 4, 2019 at 9:02 PM Randall Hauch <rh...@gmail.com> wrote:
> Sorry that I was not clear. Yes, I was suggesting the state-specific counts
> were in addition to the simple task count you originally proposed. Thanks
> for taking my suggestion into account -- the updated KIP looks great.
>
> Thanks for contributing this improvement, Cyrus!
>
> Best regards,
>
> Randall
>
> On Tue, Jun 4, 2019 at 6:35 PM Cyrus Vafadari <cy...@confluent.io> wrote:
>
> > Randall,
> >
> > I've updated the KIP to include all of your recommendations!
> >
> > Cyrus
> >
> > On Tue, Jun 4, 2019 at 2:55 PM Cyrus Vafadari <cy...@confluent.io>
> wrote:
> >
> > > Randall,
> > >
> > > I plan to update the public details section and the performance impact
> as
> > > you recommended.
> > >
> > > Regarding state-specific counts, I do agree this is a useful addition.
> > > Before I make the change, I'd like to agree that these state-specific
> > > counts should be in addition to the already-proposed total tasks count
> > > (even though might be redundant, it is robust against new/missed
> > connector
> > > states, and is a useful metric in its own right), yes?
> > >
> > > Cyrus
> > >
> > > On Tue, Jun 4, 2019 at 12:24 PM Randall Hauch <rh...@gmail.com>
> wrote:
> > >
> > >> Thanks, Cyrus -- this will be quite useful. I do have a few
> > >> comments/requests.
> > >>
> > >> Can you please be more specific about the public details about the
> > metric?
> > >> What is the MBean name on which the metric will appear? For example,
> the
> > >> AK
> > >> documentation (
> > https://kafka.apache.org/documentation/#connect_monitoring
> > >> )
> > >> defines all of the metrics an where they will appear, as does
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework
> > >> .
> > >>
> > >> Secondly, while a metric showing the total number of tasks is very
> > useful,
> > >> might it be worth considering also adding metrics for the number of
> > >> running
> > >> tasks, the number of paused tasks, and the number of failed tasks for
> a
> > >> connector. It might require using the herder's `connectorStatus(String
> > >> connectorName)` method instead, but that appears to be just as
> effective
> > >> at
> > >> using the local snapshot of the status store cache.
> > >>
> > >> Thirdly, it might be useful for the KIP to address the potential
> > >> performance impact of computing these methods. Again, IIUC, the herder
> > >> methods that the proposal mentions use the status and config stores
> > caches
> > >> only, so the impact should be negligible.
> > >>
> > >> Best regards,
> > >>
> > >> Randall
> > >>
> > >> On Sun, Jun 2, 2019 at 10:05 PM Ryanne Dolan <ry...@gmail.com>
> > >> wrote:
> > >>
> > >> > Cyrus, I agree this would be useful.
> > >> >
> > >> > Ryanne
> > >> >
> > >> > On Fri, May 31, 2019, 7:10 PM Oleksandr Diachenko <
> > >> odiachenko@apache.org>
> > >> > wrote:
> > >> >
> > >> > >
> > >> > >
> > >> > > On 2019/05/30 06:06:12, Cyrus Vafadari <cy...@confluent.io>
> wrote:
> > >> > > > Hello Dev,
> > >> > > >
> > >> > > > I'd like to start the discussion of KIP-475: New Metric to
> Measure
> > >> > Number
> > >> > > > of Tasks on a Connector.
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector
> > >> > > >
> > >> > > > The proposal is pretty straightforward -- to add a new metric to
> > >> > Connect
> > >> > > to
> > >> > > > measure the number of tasks on a Connector. Currently, we
> support
> > >> this
> > >> > on
> > >> > > > Worker level, so this KIP just adds another metric to support
> this
> > >> > > > per-connector.
> > >> > > >
> > >> > > > There is also a PR:
> > >> > > > https://github.com/apache/kafka/pull/6843
> > >> > > >
> > >> > > > Thanks,
> > >> > > >
> > >> > > > Cyrus
> > >> > > >
> > >> > >
> > >> > > Hi Cyrus,
> > >> > >
> > >> > > That sounds like a useful addition.
> > >> > >
> > >> > > Regards, Alex.
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Posted by Randall Hauch <rh...@gmail.com>.
Sorry that I was not clear. Yes, I was suggesting the state-specific counts
were in addition to the simple task count you originally proposed. Thanks
for taking my suggestion into account -- the updated KIP looks great.
Thanks for contributing this improvement, Cyrus!
Best regards,
Randall
On Tue, Jun 4, 2019 at 6:35 PM Cyrus Vafadari <cy...@confluent.io> wrote:
> Randall,
>
> I've updated the KIP to include all of your recommendations!
>
> Cyrus
>
> On Tue, Jun 4, 2019 at 2:55 PM Cyrus Vafadari <cy...@confluent.io> wrote:
>
> > Randall,
> >
> > I plan to update the public details section and the performance impact as
> > you recommended.
> >
> > Regarding state-specific counts, I do agree this is a useful addition.
> > Before I make the change, I'd like to agree that these state-specific
> > counts should be in addition to the already-proposed total tasks count
> > (even though might be redundant, it is robust against new/missed
> connector
> > states, and is a useful metric in its own right), yes?
> >
> > Cyrus
> >
> > On Tue, Jun 4, 2019 at 12:24 PM Randall Hauch <rh...@gmail.com> wrote:
> >
> >> Thanks, Cyrus -- this will be quite useful. I do have a few
> >> comments/requests.
> >>
> >> Can you please be more specific about the public details about the
> metric?
> >> What is the MBean name on which the metric will appear? For example, the
> >> AK
> >> documentation (
> https://kafka.apache.org/documentation/#connect_monitoring
> >> )
> >> defines all of the metrics an where they will appear, as does
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework
> >> .
> >>
> >> Secondly, while a metric showing the total number of tasks is very
> useful,
> >> might it be worth considering also adding metrics for the number of
> >> running
> >> tasks, the number of paused tasks, and the number of failed tasks for a
> >> connector. It might require using the herder's `connectorStatus(String
> >> connectorName)` method instead, but that appears to be just as effective
> >> at
> >> using the local snapshot of the status store cache.
> >>
> >> Thirdly, it might be useful for the KIP to address the potential
> >> performance impact of computing these methods. Again, IIUC, the herder
> >> methods that the proposal mentions use the status and config stores
> caches
> >> only, so the impact should be negligible.
> >>
> >> Best regards,
> >>
> >> Randall
> >>
> >> On Sun, Jun 2, 2019 at 10:05 PM Ryanne Dolan <ry...@gmail.com>
> >> wrote:
> >>
> >> > Cyrus, I agree this would be useful.
> >> >
> >> > Ryanne
> >> >
> >> > On Fri, May 31, 2019, 7:10 PM Oleksandr Diachenko <
> >> odiachenko@apache.org>
> >> > wrote:
> >> >
> >> > >
> >> > >
> >> > > On 2019/05/30 06:06:12, Cyrus Vafadari <cy...@confluent.io> wrote:
> >> > > > Hello Dev,
> >> > > >
> >> > > > I'd like to start the discussion of KIP-475: New Metric to Measure
> >> > Number
> >> > > > of Tasks on a Connector.
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector
> >> > > >
> >> > > > The proposal is pretty straightforward -- to add a new metric to
> >> > Connect
> >> > > to
> >> > > > measure the number of tasks on a Connector. Currently, we support
> >> this
> >> > on
> >> > > > Worker level, so this KIP just adds another metric to support this
> >> > > > per-connector.
> >> > > >
> >> > > > There is also a PR:
> >> > > > https://github.com/apache/kafka/pull/6843
> >> > > >
> >> > > > Thanks,
> >> > > >
> >> > > > Cyrus
> >> > > >
> >> > >
> >> > > Hi Cyrus,
> >> > >
> >> > > That sounds like a useful addition.
> >> > >
> >> > > Regards, Alex.
> >> > >
> >> >
> >>
> >
>
Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Posted by Cyrus Vafadari <cy...@confluent.io>.
Randall,
I've updated the KIP to include all of your recommendations!
Cyrus
On Tue, Jun 4, 2019 at 2:55 PM Cyrus Vafadari <cy...@confluent.io> wrote:
> Randall,
>
> I plan to update the public details section and the performance impact as
> you recommended.
>
> Regarding state-specific counts, I do agree this is a useful addition.
> Before I make the change, I'd like to agree that these state-specific
> counts should be in addition to the already-proposed total tasks count
> (even though might be redundant, it is robust against new/missed connector
> states, and is a useful metric in its own right), yes?
>
> Cyrus
>
> On Tue, Jun 4, 2019 at 12:24 PM Randall Hauch <rh...@gmail.com> wrote:
>
>> Thanks, Cyrus -- this will be quite useful. I do have a few
>> comments/requests.
>>
>> Can you please be more specific about the public details about the metric?
>> What is the MBean name on which the metric will appear? For example, the
>> AK
>> documentation (https://kafka.apache.org/documentation/#connect_monitoring
>> )
>> defines all of the metrics an where they will appear, as does
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework
>> .
>>
>> Secondly, while a metric showing the total number of tasks is very useful,
>> might it be worth considering also adding metrics for the number of
>> running
>> tasks, the number of paused tasks, and the number of failed tasks for a
>> connector. It might require using the herder's `connectorStatus(String
>> connectorName)` method instead, but that appears to be just as effective
>> at
>> using the local snapshot of the status store cache.
>>
>> Thirdly, it might be useful for the KIP to address the potential
>> performance impact of computing these methods. Again, IIUC, the herder
>> methods that the proposal mentions use the status and config stores caches
>> only, so the impact should be negligible.
>>
>> Best regards,
>>
>> Randall
>>
>> On Sun, Jun 2, 2019 at 10:05 PM Ryanne Dolan <ry...@gmail.com>
>> wrote:
>>
>> > Cyrus, I agree this would be useful.
>> >
>> > Ryanne
>> >
>> > On Fri, May 31, 2019, 7:10 PM Oleksandr Diachenko <
>> odiachenko@apache.org>
>> > wrote:
>> >
>> > >
>> > >
>> > > On 2019/05/30 06:06:12, Cyrus Vafadari <cy...@confluent.io> wrote:
>> > > > Hello Dev,
>> > > >
>> > > > I'd like to start the discussion of KIP-475: New Metric to Measure
>> > Number
>> > > > of Tasks on a Connector.
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector
>> > > >
>> > > > The proposal is pretty straightforward -- to add a new metric to
>> > Connect
>> > > to
>> > > > measure the number of tasks on a Connector. Currently, we support
>> this
>> > on
>> > > > Worker level, so this KIP just adds another metric to support this
>> > > > per-connector.
>> > > >
>> > > > There is also a PR:
>> > > > https://github.com/apache/kafka/pull/6843
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Cyrus
>> > > >
>> > >
>> > > Hi Cyrus,
>> > >
>> > > That sounds like a useful addition.
>> > >
>> > > Regards, Alex.
>> > >
>> >
>>
>
Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Posted by Cyrus Vafadari <cy...@confluent.io>.
Randall,
I plan to update the public details section and the performance impact as
you recommended.
Regarding state-specific counts, I do agree this is a useful addition.
Before I make the change, I'd like to agree that these state-specific
counts should be in addition to the already-proposed total tasks count
(even though might be redundant, it is robust against new/missed connector
states, and is a useful metric in its own right), yes?
Cyrus
On Tue, Jun 4, 2019 at 12:24 PM Randall Hauch <rh...@gmail.com> wrote:
> Thanks, Cyrus -- this will be quite useful. I do have a few
> comments/requests.
>
> Can you please be more specific about the public details about the metric?
> What is the MBean name on which the metric will appear? For example, the AK
> documentation (https://kafka.apache.org/documentation/#connect_monitoring)
> defines all of the metrics an where they will appear, as does
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework
> .
>
> Secondly, while a metric showing the total number of tasks is very useful,
> might it be worth considering also adding metrics for the number of running
> tasks, the number of paused tasks, and the number of failed tasks for a
> connector. It might require using the herder's `connectorStatus(String
> connectorName)` method instead, but that appears to be just as effective at
> using the local snapshot of the status store cache.
>
> Thirdly, it might be useful for the KIP to address the potential
> performance impact of computing these methods. Again, IIUC, the herder
> methods that the proposal mentions use the status and config stores caches
> only, so the impact should be negligible.
>
> Best regards,
>
> Randall
>
> On Sun, Jun 2, 2019 at 10:05 PM Ryanne Dolan <ry...@gmail.com>
> wrote:
>
> > Cyrus, I agree this would be useful.
> >
> > Ryanne
> >
> > On Fri, May 31, 2019, 7:10 PM Oleksandr Diachenko <odiachenko@apache.org
> >
> > wrote:
> >
> > >
> > >
> > > On 2019/05/30 06:06:12, Cyrus Vafadari <cy...@confluent.io> wrote:
> > > > Hello Dev,
> > > >
> > > > I'd like to start the discussion of KIP-475: New Metric to Measure
> > Number
> > > > of Tasks on a Connector.
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector
> > > >
> > > > The proposal is pretty straightforward -- to add a new metric to
> > Connect
> > > to
> > > > measure the number of tasks on a Connector. Currently, we support
> this
> > on
> > > > Worker level, so this KIP just adds another metric to support this
> > > > per-connector.
> > > >
> > > > There is also a PR:
> > > > https://github.com/apache/kafka/pull/6843
> > > >
> > > > Thanks,
> > > >
> > > > Cyrus
> > > >
> > >
> > > Hi Cyrus,
> > >
> > > That sounds like a useful addition.
> > >
> > > Regards, Alex.
> > >
> >
>
Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Posted by Randall Hauch <rh...@gmail.com>.
Thanks, Cyrus -- this will be quite useful. I do have a few
comments/requests.
Can you please be more specific about the public details about the metric?
What is the MBean name on which the metric will appear? For example, the AK
documentation (https://kafka.apache.org/documentation/#connect_monitoring)
defines all of the metrics an where they will appear, as does
https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework
.
Secondly, while a metric showing the total number of tasks is very useful,
might it be worth considering also adding metrics for the number of running
tasks, the number of paused tasks, and the number of failed tasks for a
connector. It might require using the herder's `connectorStatus(String
connectorName)` method instead, but that appears to be just as effective at
using the local snapshot of the status store cache.
Thirdly, it might be useful for the KIP to address the potential
performance impact of computing these methods. Again, IIUC, the herder
methods that the proposal mentions use the status and config stores caches
only, so the impact should be negligible.
Best regards,
Randall
On Sun, Jun 2, 2019 at 10:05 PM Ryanne Dolan <ry...@gmail.com> wrote:
> Cyrus, I agree this would be useful.
>
> Ryanne
>
> On Fri, May 31, 2019, 7:10 PM Oleksandr Diachenko <od...@apache.org>
> wrote:
>
> >
> >
> > On 2019/05/30 06:06:12, Cyrus Vafadari <cy...@confluent.io> wrote:
> > > Hello Dev,
> > >
> > > I'd like to start the discussion of KIP-475: New Metric to Measure
> Number
> > > of Tasks on a Connector.
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector
> > >
> > > The proposal is pretty straightforward -- to add a new metric to
> Connect
> > to
> > > measure the number of tasks on a Connector. Currently, we support this
> on
> > > Worker level, so this KIP just adds another metric to support this
> > > per-connector.
> > >
> > > There is also a PR:
> > > https://github.com/apache/kafka/pull/6843
> > >
> > > Thanks,
> > >
> > > Cyrus
> > >
> >
> > Hi Cyrus,
> >
> > That sounds like a useful addition.
> >
> > Regards, Alex.
> >
>
Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Posted by Ryanne Dolan <ry...@gmail.com>.
Cyrus, I agree this would be useful.
Ryanne
On Fri, May 31, 2019, 7:10 PM Oleksandr Diachenko <od...@apache.org>
wrote:
>
>
> On 2019/05/30 06:06:12, Cyrus Vafadari <cy...@confluent.io> wrote:
> > Hello Dev,
> >
> > I'd like to start the discussion of KIP-475: New Metric to Measure Number
> > of Tasks on a Connector.
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector
> >
> > The proposal is pretty straightforward -- to add a new metric to Connect
> to
> > measure the number of tasks on a Connector. Currently, we support this on
> > Worker level, so this KIP just adds another metric to support this
> > per-connector.
> >
> > There is also a PR:
> > https://github.com/apache/kafka/pull/6843
> >
> > Thanks,
> >
> > Cyrus
> >
>
> Hi Cyrus,
>
> That sounds like a useful addition.
>
> Regards, Alex.
>
Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Posted by Oleksandr Diachenko <od...@apache.org>.
On 2019/05/30 06:06:12, Cyrus Vafadari <cy...@confluent.io> wrote:
> Hello Dev,
>
> I'd like to start the discussion of KIP-475: New Metric to Measure Number
> of Tasks on a Connector.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector
>
> The proposal is pretty straightforward -- to add a new metric to Connect to
> measure the number of tasks on a Connector. Currently, we support this on
> Worker level, so this KIP just adds another metric to support this
> per-connector.
>
> There is also a PR:
> https://github.com/apache/kafka/pull/6843
>
> Thanks,
>
> Cyrus
>
Hi Cyrus,
That sounds like a useful addition.
Regards, Alex.