You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Monica Song (Jira)" <ji...@apache.org> on 2020/09/22 06:04:00 UTC
[jira] [Commented] (BEAM-10855) Add reference for appoximate unique
algorithms in java and python SDK
[ https://issues.apache.org/jira/browse/BEAM-10855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17199836#comment-17199836 ]
Monica Song commented on BEAM-10855:
------------------------------------
the algorithm is KMV (keeping k largest instead of k smallest) with correction for hash collisions akin to cardinality estimation using bloom filters
> Add reference for appoximate unique algorithms in java and python SDK
> ---------------------------------------------------------------------
>
> Key: BEAM-10855
> URL: https://issues.apache.org/jira/browse/BEAM-10855
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core, sdk-py-core
> Reporter: Monica Song
> Assignee: Monica Song
> Priority: P3
> Labels: doc_cleanup
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The java [1] and python [2] classes for ApproximateUnique should reference a paper or some other form of online documentation as a source for their algorithm.
> [1] [https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateUnique.java]
> [2] [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/stats.py]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)