You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Kamal Mittal via user <us...@flink.apache.org> on 2023/06/13 03:12:59 UTC

Flink bulk and record file source format metrices

Hello,

Using Flink record stream format file source API as below for parquet records reading.

FileSource.FileSourceBuilder<?> source = FileSource.forRecordStreamFormat(streamformat, path);
source.monitorContinuously(Duration.ofMillis(10000));

Want to log/generate metrices for corrupt records and for the same need to log flink metrices at source level in parquet reader class, is there any way to do that as right now no handle for SourceContext available?

Rgds,
Kamal

RE: Flink bulk and record file source format metrices

Posted by Kamal Mittal via user <us...@flink.apache.org>.
Thanks for the approach but few updates w.r.t my query sent  –

Parquet file is a binary file so when I said corrupt record it is complete file in itself can’t be processed right?
So it is not counting corrupt records rather counting corrupt files or splits in Flink?

From: Ken Krugler <kk...@transpac.com>
Sent: 16 June 2023 08:19 PM
To: user@flink.apache.org
Cc: Shammon FY <zj...@gmail.com>; Kamal Mittal <ka...@ericsson.com>
Subject: Re: Flink bulk and record file source format metrices

Hi Kamal,

In a similar situation, when a decoding failure happened I would generate a special record that I could then detect/filter out (and increment a counter) in a FilterFunction immediately following the source.

— Ken



On Jun 16, 2023, at 2:11 AM, Kamal Mittal via user <us...@flink.apache.org>> wrote:

Hello,

Any way-forward, please suggest.

Rgds,
Kamal

From: Kamal Mittal via user <us...@flink.apache.org>>
Sent: 15 June 2023 10:39 AM
To: Shammon FY <zj...@gmail.com>>
Cc: user@flink.apache.org<ma...@flink.apache.org>
Subject: RE: Flink bulk and record file source format metrices

Hello,

I need one counter matric for no. of corrupt records while decoding parquet records at data source level. I know where the corrupt record handling requires but due to non-existence of “SourceContext” or “RuntimeContext”, unable to do anything w.r.t metric.

It is needed similarly the way “SourceReaderBase” class maintaining one counter for no. of records emitted.

Rgds,
Kamal

From: Shammon FY <zj...@gmail.com>>
Sent: 14 June 2023 05:33 PM
To: Kamal Mittal <ka...@ericsson.com>>
Cc: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink bulk and record file source format metrices

Hi Kamal,

Can you give more information about the metris you want? In Flink each source task has one source reader which already has some metrics, you can refer to metrics doc[1] for more detailed information.

[1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/

Best,
Shammon FY

On Tue, Jun 13, 2023 at 11:13 AM Kamal Mittal via user <us...@flink.apache.org>> wrote:
Hello,

Using Flink record stream format file source API as below for parquet records reading.

FileSource.FileSourceBuilder<?> source = FileSource.forRecordStreamFormat(streamformat, path);
source.monitorContinuously(Duration.ofMillis(10000));

Want to log/generate metrices for corrupt records and for the same need to log flink metrices at source level in parquet reader class, is there any way to do that as right now no handle for SourceContext available?

Rgds,
Kamal

--------------------------
Ken Krugler
http://www.scaleunlimited.com<https://protect2.fireeye.com/v1/url?k=31323334-501cfaf3-313273af-454445554331-f6b7d29d8e15caa4&q=1&e=33678f8f-9480-45e5-aac4-a9d76bed10d0&u=http%3A%2F%2Fwww.scaleunlimited.com%2F>
Custom big data solutions
Flink, Pinot, Solr, Elasticsearch




Re: Flink bulk and record file source format metrices

Posted by Ken Krugler <kk...@transpac.com>.
Hi Kamal,

In a similar situation, when a decoding failure happened I would generate a special record that I could then detect/filter out (and increment a counter) in a FilterFunction immediately following the source.

— Ken


> On Jun 16, 2023, at 2:11 AM, Kamal Mittal via user <us...@flink.apache.org> wrote:
> 
> Hello,
>  
> Any way-forward, please suggest.
>  
> Rgds,
> Kamal
>  
> From: Kamal Mittal via user <user@flink.apache.org <ma...@flink.apache.org>> 
> Sent: 15 June 2023 10:39 AM
> To: Shammon FY <zjureel@gmail.com <ma...@gmail.com>>
> Cc: user@flink.apache.org <ma...@flink.apache.org>
> Subject: RE: Flink bulk and record file source format metrices
>  
> Hello,
>  
> I need one counter matric for no. of corrupt records while decoding parquet records at data source level. I know where the corrupt record handling requires but due to non-existence of “SourceContext” or “RuntimeContext”, unable to do anything w.r.t metric.
>  
> It is needed similarly the way “SourceReaderBase” class maintaining one counter for no. of records emitted.
>  
> Rgds,
> Kamal
>  
> From: Shammon FY <zjureel@gmail.com <ma...@gmail.com>> 
> Sent: 14 June 2023 05:33 PM
> To: Kamal Mittal <kamal.mittal@ericsson.com <ma...@ericsson.com>>
> Cc: user@flink.apache.org <ma...@flink.apache.org>
> Subject: Re: Flink bulk and record file source format metrices
>  
> Hi Kamal,
>  
> Can you give more information about the metris you want? In Flink each source task has one source reader which already has some metrics, you can refer to metrics doc[1] for more detailed information.
>  
> [1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/ <https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/>
>  
> Best,
> Shammon FY
>  
> On Tue, Jun 13, 2023 at 11:13 AM Kamal Mittal via user <user@flink.apache.org <ma...@flink.apache.org>> wrote:
> Hello,
>  
> Using Flink record stream format file source API as below for parquet records reading.
>  
> FileSource.FileSourceBuilder<?> source = FileSource.forRecordStreamFormat(streamformat, path);
> source.monitorContinuously(Duration.ofMillis(10000));
>  
> Want to log/generate metrices for corrupt records and for the same need to log flink metrices at source level in parquet reader class, is there any way to do that as right now no handle for SourceContext available?
>  
> Rgds,
> Kamal

--------------------------
Ken Krugler
http://www.scaleunlimited.com
Custom big data solutions
Flink, Pinot, Solr, Elasticsearch




RE: Flink bulk and record file source format metrices

Posted by Kamal Mittal via user <us...@flink.apache.org>.
Hello,

Any way-forward, please suggest.

Rgds,
Kamal

From: Kamal Mittal via user <us...@flink.apache.org>
Sent: 15 June 2023 10:39 AM
To: Shammon FY <zj...@gmail.com>
Cc: user@flink.apache.org
Subject: RE: Flink bulk and record file source format metrices

Hello,

I need one counter matric for no. of corrupt records while decoding parquet records at data source level. I know where the corrupt record handling requires but due to non-existence of “SourceContext” or “RuntimeContext”, unable to do anything w.r.t metric.

It is needed similarly the way “SourceReaderBase” class maintaining one counter for no. of records emitted.

Rgds,
Kamal

From: Shammon FY <zj...@gmail.com>>
Sent: 14 June 2023 05:33 PM
To: Kamal Mittal <ka...@ericsson.com>>
Cc: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink bulk and record file source format metrices

Hi Kamal,

Can you give more information about the metris you want? In Flink each source task has one source reader which already has some metrics, you can refer to metrics doc[1] for more detailed information.

[1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/

Best,
Shammon FY

On Tue, Jun 13, 2023 at 11:13 AM Kamal Mittal via user <us...@flink.apache.org>> wrote:
Hello,

Using Flink record stream format file source API as below for parquet records reading.

FileSource.FileSourceBuilder<?> source = FileSource.forRecordStreamFormat(streamformat, path);
source.monitorContinuously(Duration.ofMillis(10000));

Want to log/generate metrices for corrupt records and for the same need to log flink metrices at source level in parquet reader class, is there any way to do that as right now no handle for SourceContext available?

Rgds,
Kamal

RE: Flink bulk and record file source format metrices

Posted by Kamal Mittal via user <us...@flink.apache.org>.
Hello,

I need one counter matric for no. of corrupt records while decoding parquet records at data source level. I know where the corrupt record handling requires but due to non-existence of “SourceContext” or “RuntimeContext”, unable to do anything w.r.t metric.

It is needed similarly the way “SourceReaderBase” class maintaining one counter for no. of records emitted.

Rgds,
Kamal

From: Shammon FY <zj...@gmail.com>
Sent: 14 June 2023 05:33 PM
To: Kamal Mittal <ka...@ericsson.com>
Cc: user@flink.apache.org
Subject: Re: Flink bulk and record file source format metrices

Hi Kamal,

Can you give more information about the metris you want? In Flink each source task has one source reader which already has some metrics, you can refer to metrics doc[1] for more detailed information.

[1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/

Best,
Shammon FY

On Tue, Jun 13, 2023 at 11:13 AM Kamal Mittal via user <us...@flink.apache.org>> wrote:
Hello,

Using Flink record stream format file source API as below for parquet records reading.

FileSource.FileSourceBuilder<?> source = FileSource.forRecordStreamFormat(streamformat, path);
source.monitorContinuously(Duration.ofMillis(10000));

Want to log/generate metrices for corrupt records and for the same need to log flink metrices at source level in parquet reader class, is there any way to do that as right now no handle for SourceContext available?

Rgds,
Kamal

Re: Flink bulk and record file source format metrices

Posted by Shammon FY <zj...@gmail.com>.
Hi Kamal,

Can you give more information about the metris you want? In Flink each
source task has one source reader which already has some metrics, you can
refer to metrics doc[1] for more detailed information.

[1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/

Best,
Shammon FY

On Tue, Jun 13, 2023 at 11:13 AM Kamal Mittal via user <
user@flink.apache.org> wrote:

> Hello,
>
>
>
> Using Flink record stream format file source API as below for parquet
> records reading.
>
>
>
> FileSource.FileSourceBuilder<?> source = FileSource.
> *forRecordStreamFormat*(streamformat, path);
>
> source.monitorContinuously(Duration.*ofMillis*(10000));
>
>
>
> Want to log/generate metrices for corrupt records and for the same need to
> log flink metrices at *source level* in parquet reader class, is there
> any way to do that as right now no handle for SourceContext available?
>
>
>
> Rgds,
>
> Kamal
>