You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Niels Basjes <Ni...@basjes.nl> on 2015/12/14 16:25:18 UTC
Tiny topology shows '0' for all stats.
Hi,
I have a very small topology here.
In fact this is a thing that generates synthetic data and puts it into
Kafka.
When looking at the web UI I see that all counters (i.e. Bytes
received, Records
received, Bytes sent, Records sent) all remain 0.
I verified and I'm seeing thousands of records arriving into Kafka.
Is this a bug in Flink or am I misinterpreting the meaning of these numbers?
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: Tiny topology shows '0' for all stats.
Posted by Stephan Ewen <se...@apache.org>.
To sum this up, the web dashboard stats are Flink network stats. If you job
has no network communication, its all zero.
On Mon, Dec 14, 2015 at 5:03 PM, Ufuk Celebi <uc...@apache.org> wrote:
>
> > On 14 Dec 2015, at 16:25, Niels Basjes <Ni...@basjes.nl> wrote:
> >
> > Hi,
> >
> > I have a very small topology here.
> > In fact this is a thing that generates synthetic data and puts it into
> Kafka.
> > When looking at the web UI I see that all counters (i.e. Bytes received,
> Records received, Bytes sent, Records sent) all remain 0.
> > I verified and I'm seeing thousands of records arriving into Kafka.
> >
> > Is this a bug in Flink or am I misinterpreting the meaning of these
> numbers?
>
> Sources and sinks do not show the number of received or sent records,
> because of the internals of how these numbers are collected. I agree that
> this is confusing. Big +1 to improve this.
>
> You actually don’t see any numbers, because the operators are chained and
> hence you only have one task, which acts as both source and sink.
>
> You should see some sent and received numbers if you break up the chain
> (env.disableOperatorChaining()). Can you confirm this?
>
> – Ufuk
>
>
Re: Tiny topology shows '0' for all stats.
Posted by Niels Basjes <Ni...@basjes.nl>.
@Nick: That is what I read in the current description and comments.
On Tue, Dec 15, 2015 at 6:25 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> For my own understanding, are you suggesting the FLINK-2944 (or a subtask)
> is the appropriate place to implement exposure of metrics such as bytes,
> records in, out of Streaming sources and sinks?
>
> On Tue, Dec 15, 2015 at 5:24 AM, Niels Basjes <Ni...@basjes.nl> wrote:
>
>> Hi,
>>
>> @Ufuk: I added the env.disableOperatorChaining() and indeed now I see two
>> things on the screen and there are numbers counting what has happened.
>> @Stephan: Yes, I understand these numbers now.
>>
>> I found that this is already a jira ticket to add what I was looking for:
>> https://issues.apache.org/jira/browse/FLINK-2944
>> https://issues.apache.org/jira/browse/FLINK-3130
>>
>> Niels
>>
>>
>>
>> On Mon, Dec 14, 2015 at 5:03 PM, Ufuk Celebi <uc...@apache.org> wrote:
>>
>>>
>>> > On 14 Dec 2015, at 16:25, Niels Basjes <Ni...@basjes.nl> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I have a very small topology here.
>>> > In fact this is a thing that generates synthetic data and puts it into
>>> Kafka.
>>> > When looking at the web UI I see that all counters (i.e. Bytes
>>> received, Records received, Bytes sent, Records sent) all remain 0.
>>> > I verified and I'm seeing thousands of records arriving into Kafka.
>>> >
>>> > Is this a bug in Flink or am I misinterpreting the meaning of these
>>> numbers?
>>>
>>> Sources and sinks do not show the number of received or sent records,
>>> because of the internals of how these numbers are collected. I agree that
>>> this is confusing. Big +1 to improve this.
>>>
>>> You actually don’t see any numbers, because the operators are chained
>>> and hence you only have one task, which acts as both source and sink.
>>>
>>> You should see some sent and received numbers if you break up the chain
>>> (env.disableOperatorChaining()). Can you confirm this?
>>>
>>> – Ufuk
>>>
>>>
>>
>>
>> --
>> Best regards / Met vriendelijke groeten,
>>
>> Niels Basjes
>>
>
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: Tiny topology shows '0' for all stats.
Posted by Nick Dimiduk <nd...@gmail.com>.
For my own understanding, are you suggesting the FLINK-2944 (or a subtask)
is the appropriate place to implement exposure of metrics such as bytes,
records in, out of Streaming sources and sinks?
On Tue, Dec 15, 2015 at 5:24 AM, Niels Basjes <Ni...@basjes.nl> wrote:
> Hi,
>
> @Ufuk: I added the env.disableOperatorChaining() and indeed now I see two
> things on the screen and there are numbers counting what has happened.
> @Stephan: Yes, I understand these numbers now.
>
> I found that this is already a jira ticket to add what I was looking for:
> https://issues.apache.org/jira/browse/FLINK-2944
> https://issues.apache.org/jira/browse/FLINK-3130
>
> Niels
>
>
>
> On Mon, Dec 14, 2015 at 5:03 PM, Ufuk Celebi <uc...@apache.org> wrote:
>
>>
>> > On 14 Dec 2015, at 16:25, Niels Basjes <Ni...@basjes.nl> wrote:
>> >
>> > Hi,
>> >
>> > I have a very small topology here.
>> > In fact this is a thing that generates synthetic data and puts it into
>> Kafka.
>> > When looking at the web UI I see that all counters (i.e. Bytes
>> received, Records received, Bytes sent, Records sent) all remain 0.
>> > I verified and I'm seeing thousands of records arriving into Kafka.
>> >
>> > Is this a bug in Flink or am I misinterpreting the meaning of these
>> numbers?
>>
>> Sources and sinks do not show the number of received or sent records,
>> because of the internals of how these numbers are collected. I agree that
>> this is confusing. Big +1 to improve this.
>>
>> You actually don’t see any numbers, because the operators are chained and
>> hence you only have one task, which acts as both source and sink.
>>
>> You should see some sent and received numbers if you break up the chain
>> (env.disableOperatorChaining()). Can you confirm this?
>>
>> – Ufuk
>>
>>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
Re: Tiny topology shows '0' for all stats.
Posted by Niels Basjes <Ni...@basjes.nl>.
Hi,
@Ufuk: I added the env.disableOperatorChaining() and indeed now I see two
things on the screen and there are numbers counting what has happened.
@Stephan: Yes, I understand these numbers now.
I found that this is already a jira ticket to add what I was looking for:
https://issues.apache.org/jira/browse/FLINK-2944
https://issues.apache.org/jira/browse/FLINK-3130
Niels
On Mon, Dec 14, 2015 at 5:03 PM, Ufuk Celebi <uc...@apache.org> wrote:
>
> > On 14 Dec 2015, at 16:25, Niels Basjes <Ni...@basjes.nl> wrote:
> >
> > Hi,
> >
> > I have a very small topology here.
> > In fact this is a thing that generates synthetic data and puts it into
> Kafka.
> > When looking at the web UI I see that all counters (i.e. Bytes received,
> Records received, Bytes sent, Records sent) all remain 0.
> > I verified and I'm seeing thousands of records arriving into Kafka.
> >
> > Is this a bug in Flink or am I misinterpreting the meaning of these
> numbers?
>
> Sources and sinks do not show the number of received or sent records,
> because of the internals of how these numbers are collected. I agree that
> this is confusing. Big +1 to improve this.
>
> You actually don’t see any numbers, because the operators are chained and
> hence you only have one task, which acts as both source and sink.
>
> You should see some sent and received numbers if you break up the chain
> (env.disableOperatorChaining()). Can you confirm this?
>
> – Ufuk
>
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: Tiny topology shows '0' for all stats.
Posted by Ufuk Celebi <uc...@apache.org>.
> On 14 Dec 2015, at 16:25, Niels Basjes <Ni...@basjes.nl> wrote:
>
> Hi,
>
> I have a very small topology here.
> In fact this is a thing that generates synthetic data and puts it into Kafka.
> When looking at the web UI I see that all counters (i.e. Bytes received, Records received, Bytes sent, Records sent) all remain 0.
> I verified and I'm seeing thousands of records arriving into Kafka.
>
> Is this a bug in Flink or am I misinterpreting the meaning of these numbers?
Sources and sinks do not show the number of received or sent records, because of the internals of how these numbers are collected. I agree that this is confusing. Big +1 to improve this.
You actually don’t see any numbers, because the operators are chained and hence you only have one task, which acts as both source and sink.
You should see some sent and received numbers if you break up the chain (env.disableOperatorChaining()). Can you confirm this?
– Ufuk