You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "jszhouchong@163.com" <js...@163.com> on 2019/05/13 05:58:24 UTC
how to count kafka sink number
hi
i have a flink sql, reading record from kafka, then use table function do some transformation, then produce to kafka.
i have found that in the flink web record received of the first subTask is always 0 ,and the Records send of the last subTask is 0 as well.
i want to count how many records i consumed from the source kafka and how many records i produce to the target kafka,
how can i get these information from flink directly?
best regards
chong
Re: how to count kafka sink number
Posted by Konstantin Knauf <ko...@ververica.com>.
Hi Chong,
to my knowledge, neither Flink's built-in metrics nor the metrics of the
Kafka producer itself give you this number directly. If your sink is
chained (no serialization, no network) to another Flink operator, you could
take the numRecordsOut of this operator instead. It will tell you how many
records have been received by the sink, not how many were produced to the
Kafka cluster, though.
Best,
Konstantin
On Mon, May 13, 2019 at 7:58 AM jszhouchong@163.com <js...@163.com>
wrote:
> hi
>
> i have a flink sql, reading record from kafka, then use table function do
> some transformation, then produce to kafka.
> i have found that in the flink web record received of the first subTask
> is always 0 ,and the Records send of the last subTask is 0 as well.
>
> i want to count how many records i consumed from the source kafka and how
> many records i produce to the target kafka,
> how can i get these information from flink directly?
>
> best regards
> chong
>
>
>
>
>
--
Konstantin Knauf | Solutions Architect
+49 160 91394525
Planned Absences: 20. - 21.06.2019
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen