You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Felipe Oliveira Gutierrez (JIRA)" <ji...@apache.org> on 2019/04/24 11:54:00 UTC
[jira] [Commented] (FLINK-2147) Approximate calculation of
frequencies in data streams
[ https://issues.apache.org/jira/browse/FLINK-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825067#comment-16825067 ]
Felipe Oliveira Gutierrez commented on FLINK-2147:
--------------------------------------------------
Hi [~aljoscha], [~till.rohrmann], and @all from this thread.
I would like to learn the necessary skills to help on this issue. I have implemented the CountMinSketch [1] using a ProcessWindowFunction [2]. However, what I would like to have is a function that I can call after the window like sum(), min(), etc. I have implemented a custom operator to do the Combiner [4], but I believe that this issue is a bit more complex since it has to compute after the window. So, I was planning to base my implementation on a custom JoinFuntion as [~till.rohrmann] did in this example [3].
Where do you think I can start? Do you have any sample code that I can start to build up my own custom window operator and afterward strive to implement the CountMin-Sketch for Flink?
[1] [https://github.com/felipegutierrez/explore-flink/blob/master/src/main/java/org/sense/flink/util/CountMinSketch.java]
[2] [https://github.com/felipegutierrez/explore-flink/blob/master/src/main/java/org/sense/flink/examples/stream/MultiSensorMultiStationsReadingMqtt2.java#L69]
[3] [https://github.com/tillrohrmann/custom-join]
[4] [https://github.com/felipegutierrez/explore-flink/blob/master/src/main/java/org/sense/flink/examples/stream/MqttSensorDataCombinerByKeySkewedDAG.java#L93]
Kind Regards,
Felipe
> Approximate calculation of frequencies in data streams
> ------------------------------------------------------
>
> Key: FLINK-2147
> URL: https://issues.apache.org/jira/browse/FLINK-2147
> Project: Flink
> Issue Type: New Feature
> Components: API / DataStream
> Reporter: Gabor Gevay
> Priority: Major
> Labels: approximate, statistics
>
> Count-Min sketch is a hashing-based algorithm for approximately keeping track of the frequencies of elements in a data stream. It is described by Cormode et al. in the following paper:
> http://dimacs.rutgers.edu/~graham/pubs/papers/cmsoft.pdf
> Note that this algorithm can be conveniently implemented in a distributed way, as described in section 3.2 of the paper.
> The paper
> http://www.vldb.org/conf/2002/S10P03.pdf
> also describes algorithms for approximately keeping track of frequencies, but here the user can specify a threshold below which she is not interested in the frequency of an element. The error-bounds are also different than the Count-min sketch algorithm.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)