[jira] [Created] (PIG-3395) Large filter expression makes Pig hang

["Cheolsoo Park (JIRA)" <ji...@apache.org>]

Cheolsoo Park created PIG-3395:
----------------------------------

             Summary: Large filter expression makes Pig hang
                 Key: PIG-3395
                 URL: https://issues.apache.org/jira/browse/PIG-3395
             Project: Pig
          Issue Type: Bug
          Components: impl
            Reporter: Cheolsoo Park
            Assignee: Cheolsoo Park
             Fix For: 0.12


Currently, partition filter push down is quite costly. For example, if you have many nested or/and expressions, Pig hangs:
{code}
base = load '<partitioned table>' using MyStorage();
filt = filter base by
(dateint == 20130719 and batchid == 'merged_1' and hour IN (19,20,21,22,23))
or
(dateint == 20130720 and batchid == 'merged_1' and hour IN (0,1,2,3,4,5,6,7,8))
or
(dateint == 20130720 and batchid == 'merged_2' and hour == 7)
or
(dateint == 20130720 and batchid == 'merged_1' and hour IN (9,10,11,12,13,14,15,16,17,18,19,20,21,22,23))
or
(dateint == 20130721 and batchid == 'merged_1' and hour IN (0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23))
or
(dateint == 20130722 and batchid == 'merged_1' and hour IN (0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16));
dump filt;
{code}
Note that IN operator is converted to nested OR's by Pig parser.

Looking at the thread dump, I found it creates almost 60 stack frames and makes JVM suffer. (I will attach full stack trace.)
{code}
<repeated ...>
at org.apache.pig.newplan.PColFilterExtractor.visit(PColFilterExtractor.java:504)
at org.apache.pig.newplan.PColFilterExtractor.visit(PColFilterExtractor.java:237)
at org.apache.pig.newplan.PColFilterExtractor.visit(PColFilterExtractor.java:504)
at org.apache.pig.newplan.PColFilterExtractor.visit(PColFilterExtractor.java:214)
at org.apache.pig.newplan.PColFilterExtractor.visit(PColFilterExtractor.java:504)
at org.apache.pig.newplan.PColFilterExtractor.visit(PColFilterExtractor.java:211)
at org.apache.pig.newplan.PColFilterExtractor.visit(PColFilterExtractor.java:108)
{code}
Although the filter expression can be simplified, it seems possible to make PColFilterExtractor more efficient.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira