[jira] [Commented] (DATAFU-3) Bootstrap sum UDF

["Xiangrui Meng (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DATAFU-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871705#comment-13871705 ] 

Xiangrui Meng commented on DATAFU-3:
------------------------------------

Hi Josh,

Just want to mention that there is a UDF pair in DataFu that implements a scalable algorithm I developed for simple random sampling with replacement. Please check the javadoc at 

https://github.com/linkedin/datafu/blob/master/src/java/datafu/pig/sampling/SimpleRandomSampleWithReplacementVote.java

It guarantees the correct sample size with probability at least 99.99%, while the running time is O(n + s log(s)) and the size of the intermediate storage is O(s log(s)), where n is the size of the population and s is the desired sample size.

Best,
Xiangrui

> Bootstrap sum UDF
> -----------------
>
>                 Key: DATAFU-3
>                 URL: https://issues.apache.org/jira/browse/DATAFU-3
>             Project: DataFu
>          Issue Type: Bug
>            Reporter: Josh Wills
>
> There was a Sawzall table called bootstrapsum that I used to find handy for some of the analysis work I did at teh goog:
> http://szl.googlecode.com/svn/trunk/src/emitters/szlbootstrapsum.cc
> It would be nice to have it back again in the Hadoop ecosystem. There was a good blog post about the utility of Poisson bootstraps for random forests here:
> http://blog.cloudera.com/blog/2013/02/how-to-resample-from-a-large-data-set-in-parallel-with-r-on-hadoop/
> ...but it's useful in all sorts of nerdy stats contexts (e.g., computing confidence intervals for experiments.) I'm open to the particular structure of the function; it could either have:
> 1) A constructor that took in the number of bootstrap samples to create and then a call() method that took in a counting variable and a weighting variable, or
> 2) Three args to the call method (num samples, counting variable, and weighting variable, in some order.)
> The return type would be a bag of tuples, (index: int, sum: T) where the type of the sum would depend on the input type of the counting variable. index = 0 would always be the actual sum computed, while the rest of the indices would be numbered 1..numSamples for each of the different bootstrap samples.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)