[jira] [Comment Edited] (BEAM-12181) Implement parallelized (approximate) mode

["Brian Hulette (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-12181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17470050#comment-17470050 ] 

Brian Hulette edited comment on BEAM-12181 at 1/6/22, 5:04 PM:
---------------------------------------------------------------

I looked into this approach a little bit, it's discussed in the wikipedia articale: https://en.wikipedia.org/wiki/Mode_(statistics)#Mode_of_a_sample

{quote}In order to estimate the mode of the underlying distribution, the usual practice is to discretize the data by assigning frequency values to intervals of equal distance, as for making a histogram, effectively replacing the values by the midpoints of the intervals they are assigned to. The mode is then the value where the histogram reaches its peak. *For small or middle-sized samples the outcome of this procedure is sensitive to the choice of interval width if chosen too narrow or too wide*; typically one should have a sizable fraction of the data concentrated in a relatively small number of intervals (5 to 10), while the fraction of the data falling outside these intervals is also sizable. An alternate approach is kernel density estimation, which essentially blurs point samples to produce a continuous estimate of the probability density function which can provide an estimate of the mode.{quote}

(emphasis mine)

I think we could do this discretization in a distributed way, but how would we select the number of bins to use to minimize the error? It might be worth looking into the kernel density estimation approach.


was (Author: bhulette):
I looked into this approach a little bit, it's discussed in the wikipedia articale: https://en.wikipedia.org/wiki/Mode_(statistics)#Mode_of_a_sample

{quote}
In order to estimate the mode of the underlying distribution, the usual practice is to discretize the data by assigning frequency values to intervals of equal distance, as for making a histogram, effectively replacing the values by the midpoints of the intervals they are assigned to. The mode is then the value where the histogram reaches its peak. *For small or middle-sized samples the outcome of this procedure is sensitive to the choice of interval width if chosen too narrow or too wide*; typically one should have a sizable fraction of the data concentrated in a relatively small number of intervals (5 to 10), while the fraction of the data falling outside these intervals is also sizable. An alternate approach is kernel density estimation, which essentially blurs point samples to produce a continuous estimate of the probability density function which can provide an estimate of the mode.
{quote}

(emphasis mine)

I think we could do this discretization in a distributed way, but how would we select the number of bins to use to minimize the error? It might be worth looking into the kernel density estimation approach.

> Implement parallelized (approximate) mode
> -----------------------------------------
>
>                 Key: BEAM-12181
>                 URL: https://issues.apache.org/jira/browse/BEAM-12181
>             Project: Beam
>          Issue Type: Improvement
>          Components: dsl-dataframe, sdk-py-core
>            Reporter: Brian Hulette
>            Assignee: Svetak Vihaan Sundhar
>            Priority: P3
>              Labels: dataframe-api
>
> Currently we require Singleton partitioning to compute mode(). We should provide an option to compute approximate mode() which can be parallelized.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)