You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Stefano Bortoli <s....@gmail.com> on 2014/11/13 15:15:06 UTC
record processing counter
Hi all,
working with the dashboard I found extremely nice to have a monitor of the
tasks processing. I wonder whether it is feasible, or could be interesting,
to have a counter associated with the processing of each of the functions.
I would not mind to have a little overhead in the overall process if I
could understand where I am and be able to estimate the process duration.
:-)
what do you think?
saluti,
Stefano
Re: record processing counter
Posted by Ufuk Celebi <uc...@apache.org>.
Sure, go ahead. :)
I think following issues are related: FLINK-964, FLINK-766. There is also
an old GSoC proposal, which is related
https://github.com/stratosphere/stratosphere/wiki/GSoC-2014-Project-Proposal-Draft-by-Rajika-Kumarasiri
.
It would be cool if you could link the issues and also post the link.
<https://issues.apache.org/jira/browse/FLINK-964>
On Thu, Nov 13, 2014 at 4:00 PM, Stefano Bortoli <s....@gmail.com>
wrote:
> Hi Ufuk,
>
> thanks for the tip. I am already happy to know it will be on dashboard one
> day. Shall I open a ticket/feature request?
>
> For the moment I rely on old school note of execution time. The dataset
> will not change much, and I can have a reference once I get it to complete
> the first overall process. :-)
>
> saluti,
> Stefano
>
> 2014-11-13 15:46 GMT+01:00 Ufuk Celebi <uc...@apache.org>:
>
>> Hey Stefano,
>>
>> this is definitely possible and also not hard to count. See the
>> RecordWriter class, which emits all records to the network runtime. The
>> problem at the moment is that we don't have a central service or the like,
>> which would be able to aggregate these kinds of runtime information and
>> then (for example) have them displayed on the web interface.
>>
>> This is on the agenda and in my opinion is very important.
>>
>> If this is important for you, I could add debug log messages or so, which
>> you would then have to parse on your own. Would that be feasible as a short
>> term solution?
>>
>> On Thu, Nov 13, 2014 at 3:15 PM, Stefano Bortoli <s....@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> working with the dashboard I found extremely nice to have a monitor of
>>> the tasks processing. I wonder whether it is feasible, or could be
>>> interesting, to have a counter associated with the processing of each of
>>> the functions. I would not mind to have a little overhead in the overall
>>> process if I could understand where I am and be able to estimate the
>>> process duration. :-)
>>>
>>> what do you think?
>>>
>>> saluti,
>>> Stefano
>>>
>>
>>
>
Re: record processing counter
Posted by Stefano Bortoli <s....@gmail.com>.
Hi Ufuk,
thanks for the tip. I am already happy to know it will be on dashboard one
day. Shall I open a ticket/feature request?
For the moment I rely on old school note of execution time. The dataset
will not change much, and I can have a reference once I get it to complete
the first overall process. :-)
saluti,
Stefano
2014-11-13 15:46 GMT+01:00 Ufuk Celebi <uc...@apache.org>:
> Hey Stefano,
>
> this is definitely possible and also not hard to count. See the
> RecordWriter class, which emits all records to the network runtime. The
> problem at the moment is that we don't have a central service or the like,
> which would be able to aggregate these kinds of runtime information and
> then (for example) have them displayed on the web interface.
>
> This is on the agenda and in my opinion is very important.
>
> If this is important for you, I could add debug log messages or so, which
> you would then have to parse on your own. Would that be feasible as a short
> term solution?
>
> On Thu, Nov 13, 2014 at 3:15 PM, Stefano Bortoli <s....@gmail.com>
> wrote:
>
>> Hi all,
>>
>> working with the dashboard I found extremely nice to have a monitor of
>> the tasks processing. I wonder whether it is feasible, or could be
>> interesting, to have a counter associated with the processing of each of
>> the functions. I would not mind to have a little overhead in the overall
>> process if I could understand where I am and be able to estimate the
>> process duration. :-)
>>
>> what do you think?
>>
>> saluti,
>> Stefano
>>
>
>
Re: record processing counter
Posted by Ufuk Celebi <uc...@apache.org>.
Hey Stefano,
this is definitely possible and also not hard to count. See the
RecordWriter class, which emits all records to the network runtime. The
problem at the moment is that we don't have a central service or the like,
which would be able to aggregate these kinds of runtime information and
then (for example) have them displayed on the web interface.
This is on the agenda and in my opinion is very important.
If this is important for you, I could add debug log messages or so, which
you would then have to parse on your own. Would that be feasible as a short
term solution?
On Thu, Nov 13, 2014 at 3:15 PM, Stefano Bortoli <s....@gmail.com>
wrote:
> Hi all,
>
> working with the dashboard I found extremely nice to have a monitor of the
> tasks processing. I wonder whether it is feasible, or could be interesting,
> to have a counter associated with the processing of each of the functions.
> I would not mind to have a little overhead in the overall process if I
> could understand where I am and be able to estimate the process duration.
> :-)
>
> what do you think?
>
> saluti,
> Stefano
>