You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Koji Noguchi (JIRA)" <ji...@apache.org> on 2017/06/28 20:27:00 UTC
[jira] [Updated] (PIG-4767) Partition filter not pushed down when
filter clause references variable from another load path
[ https://issues.apache.org/jira/browse/PIG-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-4767:
------------------------------
Attachment: pig-4767-v01.patch
Skipping of optimization with filter expression containing scalar was done at PIG-1669. However, I don't think we need to apply this to {{PartitionFilterOptimizer}} and {{PredicatePushdownOptimizer}} since {{FilterExtractor}} that both optimizer depend on would keep the filter expression that contains {{ScalarExpression}}. Original LOFilter will be preserved with updated filter expression containing this scalar.
Reverted the changes in both optimizers and added a test for PartitionFilter with scalar.
(I looked at Predicate/Orc testing but didn't find an easy way to add this test.)
[~daijy], appreciate if you could take a look.
> Partition filter not pushed down when filter clause references variable from another load path
> ----------------------------------------------------------------------------------------------
>
> Key: PIG-4767
> URL: https://issues.apache.org/jira/browse/PIG-4767
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.15.0
> Reporter: Anthony Hsu
> Assignee: Koji Noguchi
> Fix For: 0.18.0
>
> Attachments: pig-4767-v01.patch
>
>
> To reproduce:
> {noformat:title=test.pig}
> a = load 'a.txt';
> a_group = group a all;
> a_count = foreach a_group generate COUNT(a) as count;
> b = load 'mytable' using org.apache.hcatalog.pig.HCatLoader();
> b = filter b by datepartition == '2015-09-01-00' and foo == a_count.count;
> dump b;
> {noformat}
> The above query ends up reading all the table partitions. If you remove the {{foo == a_count.count}} clause or replace {{a_count.count}} with a constant, then partition filtering happens properly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)