[jira] [Created] (PIG-2133) JOIN after SAMPLE appears to resample the original dataset, leads to major confusion

["Gabor Szabo (JIRA)" <ji...@apache.org>]

JOIN after SAMPLE appears to resample the original dataset, leads to major confusion
------------------------------------------------------------------------------------

                 Key: PIG-2133
                 URL: https://issues.apache.org/jira/browse/PIG-2133
             Project: Pig
          Issue Type: Bug
          Components: build
    Affects Versions: 0.8.1
         Environment: CentOS release 5.5 (Final)
            Reporter: Gabor Szabo


The following example illustrates:

data1 = LOAD ...
sampled_data1 = SAMPLE data1 0.1;
STORE sampled_data1 INTO sampled_data1.file;

data2 = LOAD ...
joined = JOIN sampled_data1 BY field1, data2 BY field2;
STORE joined INTO joined.file;

What I found is that records in joined.file DO NOT appear in sampled_data1.file, although if they could be joined with sampled_data1 in the first place, they should. The execution steps seem to indicate that the 2nd MR step generates a new sample from data1 before it joins it with data2. Therefore if the sampling rate is low we won't find most of the records of joined in sampled_data1.file .

The fix (thanks Dmitriy!): force flushing and reload the sampled data by inserting after the first STORE:

EXEC;
sampled_data1 = LOAD sampled_data1.file;


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira