You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Flavio Pompermaier <po...@okkam.it> on 2017/04/05 07:55:08 UTC
Flink UI and records sent
Hi to all,
I'm using Flink 1.2.0 and I have a job that (at some point) calls
dataset.first(1M).
Sometimes the records sent displayed by the UI are less than 1M (lik
999709).
Is it possible that the UI (or the internal Flink counters) miss some
record?
Best,
Flavio
--
Flavio Pompermaier
Development Department
OKKAM S.r.l.
Tel. +(39) 0461 1823908
Re: Flink UI and records sent
Posted by Chesnay Schepler <ch...@apache.org>.
Could you verify with a custom UDF that actually 1m records are being
produced?
Since 3 separate tasks report a consistent number of incoming/outgoing
records
I would rule out an issue in the metric system. These metrics are all
counted
separately from each other; having the same inconsistency everywhere is nigh
impossible.
Is this reproducible, and is it possible for you to provide me with the
job used?
On 05.04.2017 13:52, Flavio Pompermaier wrote:
> My job is a batch one.
> Here's an image of two different execution of the job.
> The third line is where the first(1M) is called. In the left side the
> count is what I expect, in the second is slightly less :(
>
> Inline image 1
>
> Any idea?
>
> On Wed, Apr 5, 2017 at 12:51 PM, Chesnay Schepler <chesnay@apache.org
> <ma...@apache.org>> wrote:
>
> Hey Flavio,
>
> it's unlikely that the counters skip a record.
>
> For the webUI these metrics are transported in 2 different ways:
> For running tasks they are fetched through the metric system; this
> provides no guarantee that the final count is ever displayed.
> For finished tasks the final count is stored in the ExecutionGraph
> and should show an accurate final count.
>
> So, the question is in which state your task is.
>
> Regards,
> Chesnay
>
>
> On 05.04.2017 09:55, Flavio Pompermaier wrote:
>> Hi to all,
>> I'm using Flink 1.2.0 and I have a job that (at some point) calls
>> dataset.first(1M).
>> Sometimes the records sent displayed by the UI are less than 1M
>> (lik 999709).
>> Is it possible that the UI (or the internal Flink counters) miss
>> some record?
>>
>> Best,
>> Flavio
>>
>> --
>> Flavio Pompermaier
>> Development Department
>>
>> OKKAM S.r.l.
>> Tel. +(39) 0461 1823908 <tel:+39%200461%20182%203908>
>
>
>
>
> --
> Flavio Pompermaier
> Development Department
>
> OKKAM S.r.l.
> Tel. +(39) 0461 1823908
Re: Flink UI and records sent
Posted by Flavio Pompermaier <po...@okkam.it>.
My job is a batch one.
Here's an image of two different execution of the job.
The third line is where the first(1M) is called. In the left side the count
is what I expect, in the second is slightly less :(
[image: Inline image 1]
Any idea?
On Wed, Apr 5, 2017 at 12:51 PM, Chesnay Schepler <ch...@apache.org>
wrote:
> Hey Flavio,
>
> it's unlikely that the counters skip a record.
>
> For the webUI these metrics are transported in 2 different ways:
> For running tasks they are fetched through the metric system; this
> provides no guarantee that the final count is ever displayed.
> For finished tasks the final count is stored in the ExecutionGraph and
> should show an accurate final count.
>
> So, the question is in which state your task is.
>
> Regards,
> Chesnay
>
>
> On 05.04.2017 09:55, Flavio Pompermaier wrote:
>
> Hi to all,
> I'm using Flink 1.2.0 and I have a job that (at some point) calls
> dataset.first(1M).
> Sometimes the records sent displayed by the UI are less than 1M (lik
> 999709).
> Is it possible that the UI (or the internal Flink counters) miss some
> record?
>
> Best,
> Flavio
>
> --
> Flavio Pompermaier
> Development Department
>
> OKKAM S.r.l.
> Tel. +(39) 0461 1823908 <+39%200461%20182%203908>
>
>
>
--
Flavio Pompermaier
Development Department
OKKAM S.r.l.
Tel. +(39) 0461 1823908
Re: Flink UI and records sent
Posted by Chesnay Schepler <ch...@apache.org>.
Hey Flavio,
it's unlikely that the counters skip a record.
For the webUI these metrics are transported in 2 different ways:
For running tasks they are fetched through the metric system; this
provides no guarantee that the final count is ever displayed.
For finished tasks the final count is stored in the ExecutionGraph and
should show an accurate final count.
So, the question is in which state your task is.
Regards,
Chesnay
On 05.04.2017 09:55, Flavio Pompermaier wrote:
> Hi to all,
> I'm using Flink 1.2.0 and I have a job that (at some point) calls
> dataset.first(1M).
> Sometimes the records sent displayed by the UI are less than 1M (lik
> 999709).
> Is it possible that the UI (or the internal Flink counters) miss some
> record?
>
> Best,
> Flavio
>
> --
> Flavio Pompermaier
> Development Department
>
> OKKAM S.r.l.
> Tel. +(39) 0461 1823908