[jira] Resolved: (PIG-672) bad data output from STREAM operator in trunk (regression from 0.1.1)

["Pradeep Kamath (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pradeep Kamath resolved PIG-672.
--------------------------------

    Resolution: Invalid

Closing the issue based on the last comment. Please reopen if there is still an issue. 

> bad data output from STREAM operator in trunk (regression from 0.1.1)
> ---------------------------------------------------------------------
>
>                 Key: PIG-672
>                 URL: https://issues.apache.org/jira/browse/PIG-672
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: types_branch
>         Environment: Red Hat Enterprise Linux 4 & 5
> Hadoop 0.18.2
>            Reporter: Daniel Lescohier
>            Priority: Critical
>
> In the 0.1.1 release of pig, all of the following works fine; the problem is in the trunk version.  Here's a brief intro to the workflow (details below):
>  * I have 174856784 lines of input data, each line is a unique title string.
>  * I stream the data through `sha1.py`, which outputs a sha1 hash of each input line: a string of 40 hexadecimal digits.
>  * I group on the hash, generating a count of each group, then filter on rows having a count > 1.
>  * With pig 0.1.1, it outputs all 0-byte part-* files, because all the hashes are unique.
>  * I've also verified totally outside of Hadoop, using sort and uniq, that the hashes are unique.
>  * pig trunk checkout with "last changed rev 737863" returns non-zero results; the 7 part-* files are 1.5MB.
>  * I've tracked it down to the STREAM operation (details below).
> Here's the pig-svn-trunk job that produces the hashes:
> set job.name 'title hash';
> DEFINE Cmd `sha1.py` ship('sha1.py');
> row = load '/home/danl/url_title/unique_titles';
> hashes = stream row through Cmd;
> store hashes into '/home/danl/url_title/title_hash';
> Here's the pig-0.1.1 job that produces the hashes:
> set job.name 'title hash 011';
> DEFINE Cmd `sha1.py` ship('sha1.py');
> row = load '/home/danl/url_title/unique_titles';
> hashes = stream row through Cmd;
> store hashes into '/home/danl/url_title/title_hash.011';
> Here's sha1.py:
> #!/opt/cnet-python/default-2.5/bin/python
> from sys import stdin, stdout
> from hashlib import sha1
> for line in stdin:
>     h = sha1()
>     h.update(line[:-1])
>     stdout.write("%s\n" % h.hexdigest())
> Here's the pig-svn-trunk job for finding duplicate hashes from the hashes data generated by pig-svn-trunk:
> set job.name 'h40';
> hash = load '/home/danl/url_title/title_hash';
> grouped = group hash by $0 parallel 7;
> counted = foreach grouped generate group, COUNT(hash) as cnt;
> having = filter counted by cnt > 1;
> store having into '/home/danl/url_title/title_hash_collisions/h40';
> The seven part-* files in /home/danl/url_title/title_hash_collisions/h40 are 1.5MB each.
> Here's the pig-0.1.1 job for finding duplicate hashes from the hashes data generated by pig-0.1.1:
> set job.name 'h40.011.nh';
> hash = load '/home/danl/url_title/title_hash.011';
> grouped = group hash by $0 parallel 7;
> counted = foreach grouped generate group, COUNT(hash) as cnt;
> having = filter counted by cnt > 1;
> store having into '/home/danl/url_title/title_hash_collisions/h40.011.nh';
> The seven part-* files in /home/danl/url_title/title_hash_collisions/h40.011.nh are 0KB each.
> Here's the pig-0.1.1 job for finding duplicate hashes from the hashes data generated by pig-svn-trunk:
> set job.name 'h40.011';
> hash = load '/home/danl/url_title/title_hash';
> grouped = group hash by $0 parallel 7;
> counted = foreach grouped generate group, COUNT(hash) as cnt;
> having = filter counted by cnt > 1;
> store having into '/home/danl/url_title/title_hash_collisions/h40.011';
> The seven part-* files in /home/danl/url_title/title_hash_collisions/h40.011 are 1.5MB each.
> Therefore, it's the hash data generated by pig-svn-trunk (/home/danl/url_title/title_hash) which has duplicates in it.
> Here are the first six lines of /home/danl/url_title/title_hash/part-00064.  You can see that lines five and six are duplicates.  It looks like the stream operator read the same line twice from the Python program? The job which produces the hashes is a map-only job, no reduces.
> 8f3513136b1c8b87b8b73b9d39d96555095e9cdd
> 2edb20c5a3862cc5f545ae649f1e26430a38bda4
> ca9c216629fce16b4c113c0d9fcf65f906ab5e04
> 03fe80633822215a6935bcf95305bb14adf23f18
> 03fe80633822215a6935bcf95305bb14adf23f18
> 6d324b2cd1c52f564e2a29fcbf6ae2fb83d2697c
> After narrowing it down to the stream operator in pig-svn-trunk, I decided to run the find dupes job again using pig-svn-trunk, but to first pipe the data through cat.  Cat shouldn't change the data at all, it's an identity operation.  Here's the job:
> set job.name 'h40.cat';
> DEFINE Cmd `cat`;
> row = load '/home/danl/url_title/title_hash';
> hash = stream row through Cmd;
> grouped = group hash by $0 parallel 7;
> counted = foreach grouped generate group, COUNT(hash) as cnt;
> having = filter counted by cnt > 1;
> store having into '/home/danl/url_title/title_hash_collisions/h40.cat';
> The seven part-* files in /home/danl/url_title/title_hash_collisions/h40.cat are 7.2MB each. This 'h40.cat' job should produce the same results as the 'h40' job.  The 'h40' job had part-* files of 1.5MB each, and this job 7.2MB each, so piping it through `cat` produced even more duplicates, when `cat` is not supposed to change the results at all.
> I also ran under pig-svn-trunk a 'title hash.r2' job that re-ran creating the hashes again into another directory, just to make sure it wasn't a fluke run that produced duplicate hashes. The second time around, it also produced duplicates.  Running the dupe-detection pig job under svn-0.1.1 for the hashes produced by pig-svn-trunk from the second run, I also got 1.5MB output files.
> For a final test, I ran in pig-svn-trunk the dupe-detection code on hash data generated from pig-0.1.1:
> set job.name 'h40.trk.nh';
> hash = load '/home/danl/url_title/title_hash.011';
> grouped = group hash by $0 parallel 7;
> counted = foreach grouped generate group, COUNT(hash) as cnt;
> having = filter counted by cnt > 1;
> store having into '/home/danl/url_title/title_hash_collisions/h40.trk.nh';
> The seven part-* files in /home/danl/url_title/title_hash_collisions/h40.trk.nh are 0 bytes.  It's clear that it's the stream operation running in pig-svn-trunk which is producing the duplicates.
> Here is the complete svn info of the checkout I built pig from:
> Path: .
> URL: http://svn.apache.org/repos/asf/hadoop/pig/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 737873
> Node Kind: directory
> Schedule: normal
> Last Changed Author: pradeepkth
> Last Changed Rev: 737863
> Last Changed Date: 2009-01-26 13:27:16 -0800 (Mon, 26 Jan 2009)
> When I built it, I also ran all the unit tests.
> This was all run on Hadoop 0.18.2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.