You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Rahul Raj <ra...@gmail.com> on 2017/10/05 10:57:27 UTC

Calculating metrics real time

Hi,

I have to calculate some complicated metrics like click through rate ,
click value rate and conversions on real time data using flink. But I am
not sure what functionality of flink should I use to program this because
it involves collection of some records in memory for certain time may be 1
minute and then using formula for calculating metrics on those records.

Am I correct with my approach or is there any preferred approach for such
tasks in flink? Can I use windows for doing this? Any tutorial or example
would be really great.

Rahul Raj

Re: Calculating metrics real time

Posted by Fabian Hueske <fh...@gmail.com>.

That depends on the state backend [1] that you are using.
If you use the RocksDBStateBackend, state is written to RocksDB which
persists to disk.

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.3/ops/state_backends.html

2017-10-05 14:41 GMT+02:00 Rahul Raj <ra...@gmail.com>:

> Thanks for the response. So, i am guessing windows in flink will store the
> records in memory before processing them. Correct?
>
> Rahul Raj
>
> On Oct 5, 2017 17:50, "Fabian Hueske" <fh...@gmail.com> wrote:
>
>> Hi,
>>
>> I'd suggest to have a look at the window operators [1]. For example a
>> tumbling window of 1 minute can be used to compute metrics every minute.
>> Flink's window operators are very extensible and you can implement very
>> custom logic if the predefined windows don't match your use case. In any
>> case, windows are used to collect and perform a computation on a set of
>> records.
>>
>> Best, Fabian
>>
>> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/
>> dev/windows.html
>>
>> 2017-10-05 12:57 GMT+02:00 Rahul Raj <ra...@gmail.com>:
>>
>>> Hi,
>>>
>>> I have to calculate some complicated metrics like click through rate ,
>>> click value rate and conversions on real time data using flink. But I am
>>> not sure what functionality of flink should I use to program this because
>>> it involves collection of some records in memory for certain time may be 1
>>> minute and then using formula for calculating metrics on those records.
>>>
>>> Am I correct with my approach or is there any preferred approach for
>>> such tasks in flink? Can I use windows for doing this? Any tutorial or
>>> example would be really great.
>>>
>>> Rahul Raj
>>>
>>
>>

Re: Calculating metrics real time

Posted by Rahul Raj <ra...@gmail.com>.

Thanks for the response. So, i am guessing windows in flink will store the
records in memory before processing them. Correct?

Rahul Raj

On Oct 5, 2017 17:50, "Fabian Hueske" <fh...@gmail.com> wrote:

> Hi,
>
> I'd suggest to have a look at the window operators [1]. For example a
> tumbling window of 1 minute can be used to compute metrics every minute.
> Flink's window operators are very extensible and you can implement very
> custom logic if the predefined windows don't match your use case. In any
> case, windows are used to collect and perform a computation on a set of
> records.
>
> Best, Fabian
>
> [1] https://ci.apache.org/projects/flink/flink-docs-
> release-1.3/dev/windows.html
>
> 2017-10-05 12:57 GMT+02:00 Rahul Raj <ra...@gmail.com>:
>
>> Hi,
>>
>> I have to calculate some complicated metrics like click through rate ,
>> click value rate and conversions on real time data using flink. But I am
>> not sure what functionality of flink should I use to program this because
>> it involves collection of some records in memory for certain time may be 1
>> minute and then using formula for calculating metrics on those records.
>>
>> Am I correct with my approach or is there any preferred approach for such
>> tasks in flink? Can I use windows for doing this? Any tutorial or example
>> would be really great.
>>
>> Rahul Raj
>>
>
>

Re: Calculating metrics real time

Posted by Fabian Hueske <fh...@gmail.com>.

Hi,

I'd suggest to have a look at the window operators [1]. For example a
tumbling window of 1 minute can be used to compute metrics every minute.
Flink's window operators are very extensible and you can implement very
custom logic if the predefined windows don't match your use case. In any
case, windows are used to collect and perform a computation on a set of
records.

Best, Fabian

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/windows.html

2017-10-05 12:57 GMT+02:00 Rahul Raj <ra...@gmail.com>:

> Hi,
>
> I have to calculate some complicated metrics like click through rate ,
> click value rate and conversions on real time data using flink. But I am
> not sure what functionality of flink should I use to program this because
> it involves collection of some records in memory for certain time may be 1
> minute and then using formula for calculating metrics on those records.
>
> Am I correct with my approach or is there any preferred approach for such
> tasks in flink? Can I use windows for doing this? Any tutorial or example
> would be really great.
>
> Rahul Raj
>