You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Qingsheng Ren <re...@apache.org> on 2022/10/13 17:24:03 UTC
[VOTE] Reverting sink metric name changes made in 1.15
Hi devs,
I'd like to start a vote about reverting sink metric name changes made in
1.15 considering compatibility issues. These metrics include:
- numRecordsSend -> numRecordsOut
- numRecordsSendPerSecond -> numRecordsOutPerSecond
- numBytesSend -> numBytesOut
- numBytesSendPerSecond -> numBytesOutPerSecond
- numRecordsSendError -> numRecordsOutError
which reflect the output of the sink to the external system. "send" metric
series will be kept with the same value as "out" metric series. This change
will be applied to 1.15 and 1.16. More details could be found in the
discussion thread [1].
The vote will open for at least 72 hours.
Looking forward to your feedback!
[1] https://lists.apache.org/thread/vxhty3q97s7pw2zn0jhkyd6sxwwodzbv
Best,
Qingsheng
Re: [VOTE] Reverting sink metric name changes made in 1.15
Posted by Yun Tang <my...@live.com>.
+1 and thanks for Qingsheng's driving.
BTW, shall we consider to create a ticket to ensure backward compatibility of the metric names?
Best,
Yun Tang
________________________________
From: Jing Ge <ji...@ververica.com>
Sent: Friday, October 14, 2022 16:55
To: dev@flink.apache.org <de...@flink.apache.org>
Cc: Chesnay Schepler <ch...@apache.org>
Subject: Re: [VOTE] Reverting sink metric name changes made in 1.15
+1
The voting title might cause confusion. Technically to say, it is a further
modification on top of the current status rather than a reverting after
considering the backward compatibility.
Best regards,
Jing
On Fri, Oct 14, 2022 at 10:41 AM Qingsheng Ren <re...@apache.org> wrote:
> Thanks for the reply Chesnay!
>
> I made a POC [1] just now, and I created a draft PR [2] so that it's easier
> for everyone to leave comments on it.
>
> [1] https://github.com/PatrickRen/flink/tree/FLINK-29567-POC
> [2] https://github.com/apache/flink/pull/21065
>
> Best,
> Qingsheng
>
>
> On Fri, Oct 14, 2022 at 1:56 AM Chesnay Schepler <ch...@apache.org>
> wrote:
>
> > Do we have a PoC that achieves this without re-introducing the bug where
> > the numRecordsOut was simply wrong because it counted both records
> > written to the external system and the downstream committer?
> > It's gonna be quite the dirty hack I assume.
> >
> > On 13/10/2022 19:24, Qingsheng Ren wrote:
> > > Hi devs,
> > >
> > > I'd like to start a vote about reverting sink metric name changes made
> in
> > > 1.15 considering compatibility issues. These metrics include:
> > >
> > > - numRecordsSend -> numRecordsOut
> > > - numRecordsSendPerSecond -> numRecordsOutPerSecond
> > > - numBytesSend -> numBytesOut
> > > - numBytesSendPerSecond -> numBytesOutPerSecond
> > > - numRecordsSendError -> numRecordsOutError
> > >
> > > which reflect the output of the sink to the external system. "send"
> > metric
> > > series will be kept with the same value as "out" metric series. This
> > change
> > > will be applied to 1.15 and 1.16. More details could be found in the
> > > discussion thread [1].
> > >
> > > The vote will open for at least 72 hours.
> > >
> > > Looking forward to your feedback!
> > >
> > > [1] https://lists.apache.org/thread/vxhty3q97s7pw2zn0jhkyd6sxwwodzbv
> > >
> > > Best,
> > > Qingsheng
> > >
> >
> >
>
Re: [VOTE] Reverting sink metric name changes made in 1.15
Posted by Jing Ge <ji...@ververica.com>.
+1
The voting title might cause confusion. Technically to say, it is a further
modification on top of the current status rather than a reverting after
considering the backward compatibility.
Best regards,
Jing
On Fri, Oct 14, 2022 at 10:41 AM Qingsheng Ren <re...@apache.org> wrote:
> Thanks for the reply Chesnay!
>
> I made a POC [1] just now, and I created a draft PR [2] so that it's easier
> for everyone to leave comments on it.
>
> [1] https://github.com/PatrickRen/flink/tree/FLINK-29567-POC
> [2] https://github.com/apache/flink/pull/21065
>
> Best,
> Qingsheng
>
>
> On Fri, Oct 14, 2022 at 1:56 AM Chesnay Schepler <ch...@apache.org>
> wrote:
>
> > Do we have a PoC that achieves this without re-introducing the bug where
> > the numRecordsOut was simply wrong because it counted both records
> > written to the external system and the downstream committer?
> > It's gonna be quite the dirty hack I assume.
> >
> > On 13/10/2022 19:24, Qingsheng Ren wrote:
> > > Hi devs,
> > >
> > > I'd like to start a vote about reverting sink metric name changes made
> in
> > > 1.15 considering compatibility issues. These metrics include:
> > >
> > > - numRecordsSend -> numRecordsOut
> > > - numRecordsSendPerSecond -> numRecordsOutPerSecond
> > > - numBytesSend -> numBytesOut
> > > - numBytesSendPerSecond -> numBytesOutPerSecond
> > > - numRecordsSendError -> numRecordsOutError
> > >
> > > which reflect the output of the sink to the external system. "send"
> > metric
> > > series will be kept with the same value as "out" metric series. This
> > change
> > > will be applied to 1.15 and 1.16. More details could be found in the
> > > discussion thread [1].
> > >
> > > The vote will open for at least 72 hours.
> > >
> > > Looking forward to your feedback!
> > >
> > > [1] https://lists.apache.org/thread/vxhty3q97s7pw2zn0jhkyd6sxwwodzbv
> > >
> > > Best,
> > > Qingsheng
> > >
> >
> >
>
Re: [VOTE] Reverting sink metric name changes made in 1.15
Posted by Qingsheng Ren <re...@apache.org>.
Thanks for the reply Chesnay!
I made a POC [1] just now, and I created a draft PR [2] so that it's easier
for everyone to leave comments on it.
[1] https://github.com/PatrickRen/flink/tree/FLINK-29567-POC
[2] https://github.com/apache/flink/pull/21065
Best,
Qingsheng
On Fri, Oct 14, 2022 at 1:56 AM Chesnay Schepler <ch...@apache.org> wrote:
> Do we have a PoC that achieves this without re-introducing the bug where
> the numRecordsOut was simply wrong because it counted both records
> written to the external system and the downstream committer?
> It's gonna be quite the dirty hack I assume.
>
> On 13/10/2022 19:24, Qingsheng Ren wrote:
> > Hi devs,
> >
> > I'd like to start a vote about reverting sink metric name changes made in
> > 1.15 considering compatibility issues. These metrics include:
> >
> > - numRecordsSend -> numRecordsOut
> > - numRecordsSendPerSecond -> numRecordsOutPerSecond
> > - numBytesSend -> numBytesOut
> > - numBytesSendPerSecond -> numBytesOutPerSecond
> > - numRecordsSendError -> numRecordsOutError
> >
> > which reflect the output of the sink to the external system. "send"
> metric
> > series will be kept with the same value as "out" metric series. This
> change
> > will be applied to 1.15 and 1.16. More details could be found in the
> > discussion thread [1].
> >
> > The vote will open for at least 72 hours.
> >
> > Looking forward to your feedback!
> >
> > [1] https://lists.apache.org/thread/vxhty3q97s7pw2zn0jhkyd6sxwwodzbv
> >
> > Best,
> > Qingsheng
> >
>
>
Re: [VOTE] Reverting sink metric name changes made in 1.15
Posted by Xingbo Huang <hx...@apache.org>.
+1
Best,
Xingbo
Xintong Song <to...@gmail.com> 于2022年10月14日周五 11:42写道：
> +1 (binding)
>
> Best,
>
> Xintong
>
>
>
> On Fri, Oct 14, 2022 at 11:19 AM Jark Wu <im...@gmail.com> wrote:
>
> > Thanks a lot for driving this, Qingsheng!
> >
> > +1 (binding)
> >
> >
> > Best,
> > Jark
> >
> > On Fri, 14 Oct 2022 at 10:16, Hang Ruan <ru...@gmail.com> wrote:
> >
> > > +1 from me (non-binding).
> > >
> > > The documents about these Kafka sink metrics( Flink metrics
> > > <
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/metrics//#scope
> > > >
> > > and Kafka sink monitoring
> > > <
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kafka//#monitoring-1
> > > >)
> > > do not contain the usage of the metrics like numRecordsSend by now.
> Only
> > > Kinesis Sink contains these metrics.
> > > If the document about the metrics could be found in other places,
> please
> > > correct me. ;)
> > >
> > > Best,
> > > Hang
> > >
> > > Chesnay Schepler <ch...@apache.org> 于2022年10月14日周五 01:56写道：
> > >
> > > > Do we have a PoC that achieves this without re-introducing the bug
> > where
> > > > the numRecordsOut was simply wrong because it counted both records
> > > > written to the external system and the downstream committer?
> > > > It's gonna be quite the dirty hack I assume.
> > > >
> > > > On 13/10/2022 19:24, Qingsheng Ren wrote:
> > > > > Hi devs,
> > > > >
> > > > > I'd like to start a vote about reverting sink metric name changes
> > made
> > > in
> > > > > 1.15 considering compatibility issues. These metrics include:
> > > > >
> > > > > - numRecordsSend -> numRecordsOut
> > > > > - numRecordsSendPerSecond -> numRecordsOutPerSecond
> > > > > - numBytesSend -> numBytesOut
> > > > > - numBytesSendPerSecond -> numBytesOutPerSecond
> > > > > - numRecordsSendError -> numRecordsOutError
> > > > >
> > > > > which reflect the output of the sink to the external system. "send"
> > > > metric
> > > > > series will be kept with the same value as "out" metric series.
> This
> > > > change
> > > > > will be applied to 1.15 and 1.16. More details could be found in
> the
> > > > > discussion thread [1].
> > > > >
> > > > > The vote will open for at least 72 hours.
> > > > >
> > > > > Looking forward to your feedback!
> > > > >
> > > > > [1]
> https://lists.apache.org/thread/vxhty3q97s7pw2zn0jhkyd6sxwwodzbv
> > > > >
> > > > > Best,
> > > > > Qingsheng
> > > > >
> > > >
> > > >
> > >
> >
>
Re: [VOTE] Reverting sink metric name changes made in 1.15
Posted by Xintong Song <to...@gmail.com>.
+1 (binding)
Best,
Xintong
On Fri, Oct 14, 2022 at 11:19 AM Jark Wu <im...@gmail.com> wrote:
> Thanks a lot for driving this, Qingsheng!
>
> +1 (binding)
>
>
> Best,
> Jark
>
> On Fri, 14 Oct 2022 at 10:16, Hang Ruan <ru...@gmail.com> wrote:
>
> > +1 from me (non-binding).
> >
> > The documents about these Kafka sink metrics( Flink metrics
> > <
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/metrics//#scope
> > >
> > and Kafka sink monitoring
> > <
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kafka//#monitoring-1
> > >)
> > do not contain the usage of the metrics like numRecordsSend by now. Only
> > Kinesis Sink contains these metrics.
> > If the document about the metrics could be found in other places, please
> > correct me. ;)
> >
> > Best,
> > Hang
> >
> > Chesnay Schepler <ch...@apache.org> 于2022年10月14日周五 01:56写道：
> >
> > > Do we have a PoC that achieves this without re-introducing the bug
> where
> > > the numRecordsOut was simply wrong because it counted both records
> > > written to the external system and the downstream committer?
> > > It's gonna be quite the dirty hack I assume.
> > >
> > > On 13/10/2022 19:24, Qingsheng Ren wrote:
> > > > Hi devs,
> > > >
> > > > I'd like to start a vote about reverting sink metric name changes
> made
> > in
> > > > 1.15 considering compatibility issues. These metrics include:
> > > >
> > > > - numRecordsSend -> numRecordsOut
> > > > - numRecordsSendPerSecond -> numRecordsOutPerSecond
> > > > - numBytesSend -> numBytesOut
> > > > - numBytesSendPerSecond -> numBytesOutPerSecond
> > > > - numRecordsSendError -> numRecordsOutError
> > > >
> > > > which reflect the output of the sink to the external system. "send"
> > > metric
> > > > series will be kept with the same value as "out" metric series. This
> > > change
> > > > will be applied to 1.15 and 1.16. More details could be found in the
> > > > discussion thread [1].
> > > >
> > > > The vote will open for at least 72 hours.
> > > >
> > > > Looking forward to your feedback!
> > > >
> > > > [1] https://lists.apache.org/thread/vxhty3q97s7pw2zn0jhkyd6sxwwodzbv
> > > >
> > > > Best,
> > > > Qingsheng
> > > >
> > >
> > >
> >
>
Re: [VOTE] Reverting sink metric name changes made in 1.15
Posted by Jark Wu <im...@gmail.com>.
Thanks a lot for driving this, Qingsheng!
+1 (binding)
Best,
Jark
On Fri, 14 Oct 2022 at 10:16, Hang Ruan <ru...@gmail.com> wrote:
> +1 from me (non-binding).
>
> The documents about these Kafka sink metrics( Flink metrics
> <
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/metrics//#scope
> >
> and Kafka sink monitoring
> <
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kafka//#monitoring-1
> >)
> do not contain the usage of the metrics like numRecordsSend by now. Only
> Kinesis Sink contains these metrics.
> If the document about the metrics could be found in other places, please
> correct me. ;)
>
> Best,
> Hang
>
> Chesnay Schepler <ch...@apache.org> 于2022年10月14日周五 01:56写道：
>
> > Do we have a PoC that achieves this without re-introducing the bug where
> > the numRecordsOut was simply wrong because it counted both records
> > written to the external system and the downstream committer?
> > It's gonna be quite the dirty hack I assume.
> >
> > On 13/10/2022 19:24, Qingsheng Ren wrote:
> > > Hi devs,
> > >
> > > I'd like to start a vote about reverting sink metric name changes made
> in
> > > 1.15 considering compatibility issues. These metrics include:
> > >
> > > - numRecordsSend -> numRecordsOut
> > > - numRecordsSendPerSecond -> numRecordsOutPerSecond
> > > - numBytesSend -> numBytesOut
> > > - numBytesSendPerSecond -> numBytesOutPerSecond
> > > - numRecordsSendError -> numRecordsOutError
> > >
> > > which reflect the output of the sink to the external system. "send"
> > metric
> > > series will be kept with the same value as "out" metric series. This
> > change
> > > will be applied to 1.15 and 1.16. More details could be found in the
> > > discussion thread [1].
> > >
> > > The vote will open for at least 72 hours.
> > >
> > > Looking forward to your feedback!
> > >
> > > [1] https://lists.apache.org/thread/vxhty3q97s7pw2zn0jhkyd6sxwwodzbv
> > >
> > > Best,
> > > Qingsheng
> > >
> >
> >
>
Re: [VOTE] Reverting sink metric name changes made in 1.15
Posted by Hang Ruan <ru...@gmail.com>.
+1 from me (non-binding).
The documents about these Kafka sink metrics( Flink metrics
<https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/metrics//#scope>
and Kafka sink monitoring
<https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kafka//#monitoring-1>)
do not contain the usage of the metrics like numRecordsSend by now. Only
Kinesis Sink contains these metrics.
If the document about the metrics could be found in other places, please
correct me. ;)
Best,
Hang
Chesnay Schepler <ch...@apache.org> 于2022年10月14日周五 01:56写道：
> Do we have a PoC that achieves this without re-introducing the bug where
> the numRecordsOut was simply wrong because it counted both records
> written to the external system and the downstream committer?
> It's gonna be quite the dirty hack I assume.
>
> On 13/10/2022 19:24, Qingsheng Ren wrote:
> > Hi devs,
> >
> > I'd like to start a vote about reverting sink metric name changes made in
> > 1.15 considering compatibility issues. These metrics include:
> >
> > - numRecordsSend -> numRecordsOut
> > - numRecordsSendPerSecond -> numRecordsOutPerSecond
> > - numBytesSend -> numBytesOut
> > - numBytesSendPerSecond -> numBytesOutPerSecond
> > - numRecordsSendError -> numRecordsOutError
> >
> > which reflect the output of the sink to the external system. "send"
> metric
> > series will be kept with the same value as "out" metric series. This
> change
> > will be applied to 1.15 and 1.16. More details could be found in the
> > discussion thread [1].
> >
> > The vote will open for at least 72 hours.
> >
> > Looking forward to your feedback!
> >
> > [1] https://lists.apache.org/thread/vxhty3q97s7pw2zn0jhkyd6sxwwodzbv
> >
> > Best,
> > Qingsheng
> >
>
>
Re: [VOTE] Reverting sink metric name changes made in 1.15
Posted by Chesnay Schepler <ch...@apache.org>.
Do we have a PoC that achieves this without re-introducing the bug where
the numRecordsOut was simply wrong because it counted both records
written to the external system and the downstream committer?
It's gonna be quite the dirty hack I assume.
On 13/10/2022 19:24, Qingsheng Ren wrote:
> Hi devs,
>
> I'd like to start a vote about reverting sink metric name changes made in
> 1.15 considering compatibility issues. These metrics include:
>
> - numRecordsSend -> numRecordsOut
> - numRecordsSendPerSecond -> numRecordsOutPerSecond
> - numBytesSend -> numBytesOut
> - numBytesSendPerSecond -> numBytesOutPerSecond
> - numRecordsSendError -> numRecordsOutError
>
> which reflect the output of the sink to the external system. "send" metric
> series will be kept with the same value as "out" metric series. This change
> will be applied to 1.15 and 1.16. More details could be found in the
> discussion thread [1].
>
> The vote will open for at least 72 hours.
>
> Looking forward to your feedback!
>
> [1] https://lists.apache.org/thread/vxhty3q97s7pw2zn0jhkyd6sxwwodzbv
>
> Best,
> Qingsheng
>