You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by J Rivers <jr...@gmail.com> on 2021/11/01 20:19:20 UTC
Re: [VOTE] KIP-714: Client Metrics and Observability
+1
Thank you for the KIP!
Our organization runs kafka at large scale in a multi-tenant configuration.
We actually have many other enterprises connecting up to our system to
retrieve stream data. These feeds vary greatly in volume and velocity. The
peak rates are a multiplicative factor of the nominal. There is extreme
skew in our datasets in a number of ways.
We don't have time to work with every new internal/external client to tune
their feeds. They need to be able to take one of the many kafka clients and
go off to the races.
Being able to retrieve client metrics would be invaluable here as it's hard
and time consuming to communicate out of the enterprise walls.
This KIP is important to us to expand the use of our datasets internally
and outside the borders of the enterprise. Our clients like the performance
and data safeties related to the kafka connection. The observability has
been a problem...
Jonathan Rivers
jriversdev@gmail.com
On Mon, Oct 18, 2021 at 11:56 PM Ryanne Dolan <ry...@gmail.com> wrote:
> -1
>
> Ryanne
>
> On Mon, Oct 18, 2021, 4:30 AM Magnus Edenhill <ma...@edenhill.se> wrote:
>
> > Hi all,
> >
> > I'd like to start a vote on KIP-714.
> > https://cwiki.apache.org/confluence/x/2xRRCg
> >
> > Discussion thread:
> > https://www.mail-archive.com/dev@kafka.apache.org/msg119000.html
> >
> > Thanks,
> > Magnus
> >
>
Re: [VOTE] KIP-714: Client Metrics and Observability
Posted by Jason Gustafson <ja...@confluent.io.INVALID>.
+1 Thanks Magnus!
On Tue, May 17, 2022 at 5:43 AM Magnus Edenhill <ma...@edenhill.se> wrote:
> Hey all,
>
> It's that time of year again where we re-restart this vote thread after
> some additional
> discussions on the disco thread and minor adjustments&clarifications to the
> KIP.
>
> We're currently at +5 (non-binding) and -1 (non-binding) votes.
>
> Please cast your votes, people.
>
>
> Thanks,
> Magnus
>
>
> Den tors 3 mars 2022 kl 15:39 skrev Julien Chanaud <
> chanaud.julien@gmail.com
> >:
>
> > +1
> > As a member of a team which operates several Kafka clusters, I am
> > unequipped when it comes to troubleshooting issues with project teams
> > that did not understand the importance of configuring client-side
> > monitoring.
> > Kafka represents a fraction of their work and they don't have enough
> > experience, time or interest in trying to understand the meaning behind
> > every metric.
> >
> > I stand 100% behind what Colin stated back in June in the Discuss thread
> :
> >
> > > Magnus and I explained a few times the reasons why it does matter.
> Within
> > > most organizations, there are usually several teams using clients,
> which
> > > are separate from the team which maintains the Kafka cluster. The Kafka
> > > team has the Kafka experts, which makes it the best place to centralize
> > > collecting and analyzing Kafka metrics.
> >
> >
> > Thanks for this KIP.
> >
> > Le mer. 26 janv. 2022 à 16:01, riferrei@riferrei.com <
> > riferrei@riferrei.com>
> > a écrit :
> >
> > > +1
> > >
> > > I think this KIP solves a problem that has been around for some time
> with
> > > Kafka deployments, which is the ability to assess the current state of
> a
> > > Kafka architecture but looking at the whole picture. I also share other
> > > folks' concerns regarding adding runtime dependencies to the clients;
> > this
> > > may be problematic for large deployments. Still, I think it is worth
> > > refactoring.
> > >
> > > IMHO, it is a fair trade-off.
> > >
> > > — Ricardo
> > >
> > > > On Jan 26, 2022, at 9:34 AM, Magnus Edenhill <ma...@edenhill.se>
> > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > it's been a while and there's been some more discussions of the KIP
> > which
> > > > have been
> > > > addressed on the KIP page.
> > > >
> > > > I think it's a good time to revive this vote thread and get things
> > > moving.
> > > >
> > > > We're currently at +3 (non-binding) and -1 (non-binding) votes.
> > > >
> > > > Regards,
> > > > Magnus
> > > >
> > > >
> > > > Den mån 1 nov. 2021 kl 21:19 skrev J Rivers <jr...@gmail.com>:
> > > >
> > > >> +1
> > > >>
> > > >> Thank you for the KIP!
> > > >>
> > > >> Our organization runs kafka at large scale in a multi-tenant
> > > configuration.
> > > >> We actually have many other enterprises connecting up to our system
> to
> > > >> retrieve stream data. These feeds vary greatly in volume and
> velocity.
> > > The
> > > >> peak rates are a multiplicative factor of the nominal. There is
> > extreme
> > > >> skew in our datasets in a number of ways.
> > > >>
> > > >> We don't have time to work with every new internal/external client
> to
> > > tune
> > > >> their feeds. They need to be able to take one of the many kafka
> > clients
> > > and
> > > >> go off to the races.
> > > >>
> > > >> Being able to retrieve client metrics would be invaluable here as
> it's
> > > hard
> > > >> and time consuming to communicate out of the enterprise walls.
> > > >>
> > > >> This KIP is important to us to expand the use of our datasets
> > internally
> > > >> and outside the borders of the enterprise. Our clients like the
> > > performance
> > > >> and data safeties related to the kafka connection. The observability
> > has
> > > >> been a problem...
> > > >>
> > > >> Jonathan Rivers
> > > >> jriversdev@gmail.com
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Mon, Oct 18, 2021 at 11:56 PM Ryanne Dolan <
> ryannedolan@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> -1
> > > >>>
> > > >>> Ryanne
> > > >>>
> > > >>> On Mon, Oct 18, 2021, 4:30 AM Magnus Edenhill <ma...@edenhill.se>
> > > >> wrote:
> > > >>>
> > > >>>> Hi all,
> > > >>>>
> > > >>>> I'd like to start a vote on KIP-714.
> > > >>>> https://cwiki.apache.org/confluence/x/2xRRCg
> > > >>>>
> > > >>>> Discussion thread:
> > > >>>> https://www.mail-archive.com/dev@kafka.apache.org/msg119000.html
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Magnus
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> >
>
Re: [VOTE] KIP-714: Client Metrics and Observability
Posted by Magnus Edenhill <ma...@edenhill.se>.
Hey all,
It's that time of year again where we re-restart this vote thread after
some additional
discussions on the disco thread and minor adjustments&clarifications to the
KIP.
We're currently at +5 (non-binding) and -1 (non-binding) votes.
Please cast your votes, people.
Thanks,
Magnus
Den tors 3 mars 2022 kl 15:39 skrev Julien Chanaud <chanaud.julien@gmail.com
>:
> +1
> As a member of a team which operates several Kafka clusters, I am
> unequipped when it comes to troubleshooting issues with project teams
> that did not understand the importance of configuring client-side
> monitoring.
> Kafka represents a fraction of their work and they don't have enough
> experience, time or interest in trying to understand the meaning behind
> every metric.
>
> I stand 100% behind what Colin stated back in June in the Discuss thread :
>
> > Magnus and I explained a few times the reasons why it does matter. Within
> > most organizations, there are usually several teams using clients, which
> > are separate from the team which maintains the Kafka cluster. The Kafka
> > team has the Kafka experts, which makes it the best place to centralize
> > collecting and analyzing Kafka metrics.
>
>
> Thanks for this KIP.
>
> Le mer. 26 janv. 2022 à 16:01, riferrei@riferrei.com <
> riferrei@riferrei.com>
> a écrit :
>
> > +1
> >
> > I think this KIP solves a problem that has been around for some time with
> > Kafka deployments, which is the ability to assess the current state of a
> > Kafka architecture but looking at the whole picture. I also share other
> > folks' concerns regarding adding runtime dependencies to the clients;
> this
> > may be problematic for large deployments. Still, I think it is worth
> > refactoring.
> >
> > IMHO, it is a fair trade-off.
> >
> > — Ricardo
> >
> > > On Jan 26, 2022, at 9:34 AM, Magnus Edenhill <ma...@edenhill.se>
> wrote:
> > >
> > > Hi all,
> > >
> > > it's been a while and there's been some more discussions of the KIP
> which
> > > have been
> > > addressed on the KIP page.
> > >
> > > I think it's a good time to revive this vote thread and get things
> > moving.
> > >
> > > We're currently at +3 (non-binding) and -1 (non-binding) votes.
> > >
> > > Regards,
> > > Magnus
> > >
> > >
> > > Den mån 1 nov. 2021 kl 21:19 skrev J Rivers <jr...@gmail.com>:
> > >
> > >> +1
> > >>
> > >> Thank you for the KIP!
> > >>
> > >> Our organization runs kafka at large scale in a multi-tenant
> > configuration.
> > >> We actually have many other enterprises connecting up to our system to
> > >> retrieve stream data. These feeds vary greatly in volume and velocity.
> > The
> > >> peak rates are a multiplicative factor of the nominal. There is
> extreme
> > >> skew in our datasets in a number of ways.
> > >>
> > >> We don't have time to work with every new internal/external client to
> > tune
> > >> their feeds. They need to be able to take one of the many kafka
> clients
> > and
> > >> go off to the races.
> > >>
> > >> Being able to retrieve client metrics would be invaluable here as it's
> > hard
> > >> and time consuming to communicate out of the enterprise walls.
> > >>
> > >> This KIP is important to us to expand the use of our datasets
> internally
> > >> and outside the borders of the enterprise. Our clients like the
> > performance
> > >> and data safeties related to the kafka connection. The observability
> has
> > >> been a problem...
> > >>
> > >> Jonathan Rivers
> > >> jriversdev@gmail.com
> > >>
> > >>
> > >>
> > >>
> > >> On Mon, Oct 18, 2021 at 11:56 PM Ryanne Dolan <ry...@gmail.com>
> > >> wrote:
> > >>
> > >>> -1
> > >>>
> > >>> Ryanne
> > >>>
> > >>> On Mon, Oct 18, 2021, 4:30 AM Magnus Edenhill <ma...@edenhill.se>
> > >> wrote:
> > >>>
> > >>>> Hi all,
> > >>>>
> > >>>> I'd like to start a vote on KIP-714.
> > >>>> https://cwiki.apache.org/confluence/x/2xRRCg
> > >>>>
> > >>>> Discussion thread:
> > >>>> https://www.mail-archive.com/dev@kafka.apache.org/msg119000.html
> > >>>>
> > >>>> Thanks,
> > >>>> Magnus
> > >>>>
> > >>>
> > >>
> >
> >
>
Re: [VOTE] KIP-714: Client Metrics and Observability
Posted by Julien Chanaud <ch...@gmail.com>.
+1
As a member of a team which operates several Kafka clusters, I am
unequipped when it comes to troubleshooting issues with project teams
that did not understand the importance of configuring client-side
monitoring.
Kafka represents a fraction of their work and they don't have enough
experience, time or interest in trying to understand the meaning behind
every metric.
I stand 100% behind what Colin stated back in June in the Discuss thread :
> Magnus and I explained a few times the reasons why it does matter. Within
> most organizations, there are usually several teams using clients, which
> are separate from the team which maintains the Kafka cluster. The Kafka
> team has the Kafka experts, which makes it the best place to centralize
> collecting and analyzing Kafka metrics.
Thanks for this KIP.
Le mer. 26 janv. 2022 à 16:01, riferrei@riferrei.com <ri...@riferrei.com>
a écrit :
> +1
>
> I think this KIP solves a problem that has been around for some time with
> Kafka deployments, which is the ability to assess the current state of a
> Kafka architecture but looking at the whole picture. I also share other
> folks' concerns regarding adding runtime dependencies to the clients; this
> may be problematic for large deployments. Still, I think it is worth
> refactoring.
>
> IMHO, it is a fair trade-off.
>
> — Ricardo
>
> > On Jan 26, 2022, at 9:34 AM, Magnus Edenhill <ma...@edenhill.se> wrote:
> >
> > Hi all,
> >
> > it's been a while and there's been some more discussions of the KIP which
> > have been
> > addressed on the KIP page.
> >
> > I think it's a good time to revive this vote thread and get things
> moving.
> >
> > We're currently at +3 (non-binding) and -1 (non-binding) votes.
> >
> > Regards,
> > Magnus
> >
> >
> > Den mån 1 nov. 2021 kl 21:19 skrev J Rivers <jr...@gmail.com>:
> >
> >> +1
> >>
> >> Thank you for the KIP!
> >>
> >> Our organization runs kafka at large scale in a multi-tenant
> configuration.
> >> We actually have many other enterprises connecting up to our system to
> >> retrieve stream data. These feeds vary greatly in volume and velocity.
> The
> >> peak rates are a multiplicative factor of the nominal. There is extreme
> >> skew in our datasets in a number of ways.
> >>
> >> We don't have time to work with every new internal/external client to
> tune
> >> their feeds. They need to be able to take one of the many kafka clients
> and
> >> go off to the races.
> >>
> >> Being able to retrieve client metrics would be invaluable here as it's
> hard
> >> and time consuming to communicate out of the enterprise walls.
> >>
> >> This KIP is important to us to expand the use of our datasets internally
> >> and outside the borders of the enterprise. Our clients like the
> performance
> >> and data safeties related to the kafka connection. The observability has
> >> been a problem...
> >>
> >> Jonathan Rivers
> >> jriversdev@gmail.com
> >>
> >>
> >>
> >>
> >> On Mon, Oct 18, 2021 at 11:56 PM Ryanne Dolan <ry...@gmail.com>
> >> wrote:
> >>
> >>> -1
> >>>
> >>> Ryanne
> >>>
> >>> On Mon, Oct 18, 2021, 4:30 AM Magnus Edenhill <ma...@edenhill.se>
> >> wrote:
> >>>
> >>>> Hi all,
> >>>>
> >>>> I'd like to start a vote on KIP-714.
> >>>> https://cwiki.apache.org/confluence/x/2xRRCg
> >>>>
> >>>> Discussion thread:
> >>>> https://www.mail-archive.com/dev@kafka.apache.org/msg119000.html
> >>>>
> >>>> Thanks,
> >>>> Magnus
> >>>>
> >>>
> >>
>
>
Re: [VOTE] KIP-714: Client Metrics and Observability
Posted by "riferrei@riferrei.com" <ri...@riferrei.com>.
+1
I think this KIP solves a problem that has been around for some time with Kafka deployments, which is the ability to assess the current state of a Kafka architecture but looking at the whole picture. I also share other folks' concerns regarding adding runtime dependencies to the clients; this may be problematic for large deployments. Still, I think it is worth refactoring.
IMHO, it is a fair trade-off.
— Ricardo
> On Jan 26, 2022, at 9:34 AM, Magnus Edenhill <ma...@edenhill.se> wrote:
>
> Hi all,
>
> it's been a while and there's been some more discussions of the KIP which
> have been
> addressed on the KIP page.
>
> I think it's a good time to revive this vote thread and get things moving.
>
> We're currently at +3 (non-binding) and -1 (non-binding) votes.
>
> Regards,
> Magnus
>
>
> Den mån 1 nov. 2021 kl 21:19 skrev J Rivers <jr...@gmail.com>:
>
>> +1
>>
>> Thank you for the KIP!
>>
>> Our organization runs kafka at large scale in a multi-tenant configuration.
>> We actually have many other enterprises connecting up to our system to
>> retrieve stream data. These feeds vary greatly in volume and velocity. The
>> peak rates are a multiplicative factor of the nominal. There is extreme
>> skew in our datasets in a number of ways.
>>
>> We don't have time to work with every new internal/external client to tune
>> their feeds. They need to be able to take one of the many kafka clients and
>> go off to the races.
>>
>> Being able to retrieve client metrics would be invaluable here as it's hard
>> and time consuming to communicate out of the enterprise walls.
>>
>> This KIP is important to us to expand the use of our datasets internally
>> and outside the borders of the enterprise. Our clients like the performance
>> and data safeties related to the kafka connection. The observability has
>> been a problem...
>>
>> Jonathan Rivers
>> jriversdev@gmail.com
>>
>>
>>
>>
>> On Mon, Oct 18, 2021 at 11:56 PM Ryanne Dolan <ry...@gmail.com>
>> wrote:
>>
>>> -1
>>>
>>> Ryanne
>>>
>>> On Mon, Oct 18, 2021, 4:30 AM Magnus Edenhill <ma...@edenhill.se>
>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I'd like to start a vote on KIP-714.
>>>> https://cwiki.apache.org/confluence/x/2xRRCg
>>>>
>>>> Discussion thread:
>>>> https://www.mail-archive.com/dev@kafka.apache.org/msg119000.html
>>>>
>>>> Thanks,
>>>> Magnus
>>>>
>>>
>>
Re: [VOTE] KIP-714: Client Metrics and Observability
Posted by Magnus Edenhill <ma...@edenhill.se>.
Hi all,
it's been a while and there's been some more discussions of the KIP which
have been
addressed on the KIP page.
I think it's a good time to revive this vote thread and get things moving.
We're currently at +3 (non-binding) and -1 (non-binding) votes.
Regards,
Magnus
Den mån 1 nov. 2021 kl 21:19 skrev J Rivers <jr...@gmail.com>:
> +1
>
> Thank you for the KIP!
>
> Our organization runs kafka at large scale in a multi-tenant configuration.
> We actually have many other enterprises connecting up to our system to
> retrieve stream data. These feeds vary greatly in volume and velocity. The
> peak rates are a multiplicative factor of the nominal. There is extreme
> skew in our datasets in a number of ways.
>
> We don't have time to work with every new internal/external client to tune
> their feeds. They need to be able to take one of the many kafka clients and
> go off to the races.
>
> Being able to retrieve client metrics would be invaluable here as it's hard
> and time consuming to communicate out of the enterprise walls.
>
> This KIP is important to us to expand the use of our datasets internally
> and outside the borders of the enterprise. Our clients like the performance
> and data safeties related to the kafka connection. The observability has
> been a problem...
>
> Jonathan Rivers
> jriversdev@gmail.com
>
>
>
>
> On Mon, Oct 18, 2021 at 11:56 PM Ryanne Dolan <ry...@gmail.com>
> wrote:
>
> > -1
> >
> > Ryanne
> >
> > On Mon, Oct 18, 2021, 4:30 AM Magnus Edenhill <ma...@edenhill.se>
> wrote:
> >
> > > Hi all,
> > >
> > > I'd like to start a vote on KIP-714.
> > > https://cwiki.apache.org/confluence/x/2xRRCg
> > >
> > > Discussion thread:
> > > https://www.mail-archive.com/dev@kafka.apache.org/msg119000.html
> > >
> > > Thanks,
> > > Magnus
> > >
> >
>