You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2017/05/24 23:04:07 UTC
[jira] [Updated] (PIG-4910) Assert wrongly pushed up in optimizer
[ https://issues.apache.org/jira/browse/PIG-4910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rohini Palaniswamy updated PIG-4910:
------------------------------------
Fix Version/s: (was: 0.17.0)
0.18.0
> Assert wrongly pushed up in optimizer
> -------------------------------------
>
> Key: PIG-4910
> URL: https://issues.apache.org/jira/browse/PIG-4910
> Project: Pig
> Issue Type: Bug
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.18.0
>
>
> The following script fail:
> {code}
> TEST_DATA = LOAD 'input' USING PigStorage() AS (c1:int);
> GR = FOREACH (GROUP TEST_DATA BY c1) GENERATE group as c1, COUNT_STAR(TEST_DATA) as count, TEST_DATA;
> ROWS_WITH_C1_EQUALS_ZERO = FILTER GR BY count > 1L;
> ROWS_WITH_C1_EQUALS_ZERO_FLATTENED = FOREACH ROWS_WITH_C1_EQUALS_ZERO GENERATE FLATTEN($0);
> -- Assert shouldn't fail as it should be applied after group by but because assert is getting pushed to mapper, it is failing.
> ASSERT ROWS_WITH_C1_EQUALS_ZERO_FLATTENED BY c1 == 0, 'Should have never seen this message, assert has a bug.';
> DUMP ROWS_WITH_C1_EQUALS_ZERO_FLATTENED;
> {code}
> input:
> 0
> 0
> 1
> The reason is assert is pushed before FILTER:
> {code}
> ROWS_WITH_C1_EQUALS_ZERO_FLATTENED: (Name: LOStore Schema: c1#14:int)
> |
> |---ROWS_WITH_C1_EQUALS_ZERO_FLATTENED: (Name: LOForEach Schema: c1#14:int)
> | |
> | (Name: LOGenerate[true] Schema: c1#14:int)ColumnPrune:InputUids=[14]ColumnPrune:OutputUids=[14]
> | | |
> | | c1:(Name: Project Type: int Uid: 14 Input: 0 Column: (*))
> | |
> | |---(Name: LOInnerLoad[0] Schema: c1#14:int)
> |
> |---ROWS_WITH_C1_EQUALS_ZERO: (Name: LOFilter Schema: c1#14:int,count#31:long,ROWS_WITH_C1_EQUALS_ZERO_FLATTENED#29:bag{#30:tuple(c1#14:int)})
> | |
> | (Name: GreaterThan Type: boolean Uid: 33)
> | |
> | |---count:(Name: Project Type: long Uid: 31 Input: 0 Column: 1)
> | |
> | |---(Name: Constant Type: long Uid: 32)
> |
> |---GR: (Name: LOForEach Schema: c1#14:int,count#31:long,ROWS_WITH_C1_EQUALS_ZERO_FLATTENED#29:bag{#30:tuple(c1#14:int)})
> | |
> | (Name: LOGenerate[false,false,false] Schema: c1#14:int,count#31:long,ROWS_WITH_C1_EQUALS_ZERO_FLATTENED#29:bag{#30:tuple(c1#14:int)})ColumnPrune:InputUids=[29, 14]ColumnPrune:OutputUids=[14, 31]
> | | |
> | | group:(Name: Project Type: int Uid: 14 Input: 0 Column: (*))
> | | |
> | | (Name: UserFunc(org.apache.pig.builtin.COUNT_STAR) Type: long Uid: 31)
> | | |
> | | |---ROWS_WITH_C1_EQUALS_ZERO_FLATTENED:(Name: Project Type: bag Uid: 29 Input: 1 Column: (*))
> | | |
> | | ROWS_WITH_C1_EQUALS_ZERO_FLATTENED:(Name: Project Type: bag Uid: 29 Input: 2 Column: (*))
> | |
> | |---(Name: LOInnerLoad[0] Schema: group#14:int)
> | |
> | |---ROWS_WITH_C1_EQUALS_ZERO_FLATTENED: (Name: LOInnerLoad[1] Schema: c1#14:int)
> | |
> | |---ROWS_WITH_C1_EQUALS_ZERO_FLATTENED: (Name: LOInnerLoad[1] Schema: c1#14:int)
> |
> |---1-3: (Name: LOCogroup Schema: group#14:int,ROWS_WITH_C1_EQUALS_ZERO_FLATTENED#29:bag{#44:tuple(c1#14:int)})
> | |
> | c1:(Name: Project Type: int Uid: 14 Input: 0 Column: 0)
> |
> |---ROWS_WITH_C1_EQUALS_ZERO_FLATTENED: (Name: LOFilter Schema: c1#14:int)
> | |
> | (Name: UserFunc(org.apache.pig.builtin.Assert) Type: boolean Uid: 40)
> | |
> | |---(Name: BinCond Type: boolean Uid: 38)
> | | |
> | | |---(Name: Equal Type: boolean Uid: 35)
> | | | |
> | | | |---c1:(Name: Project Type: int Uid: 14 Input: 0 Column: 0)
> | | | |
> | | | |---(Name: Constant Type: int Uid: 34)
> | | |
> | | |---(Name: Constant Type: boolean Uid: 36)
> | | |
> | | |---(Name: Constant Type: boolean Uid: 37)
> | |
> | |---(Name: Constant Type: chararray Uid: 39)
> |
> |---TEST_DATA: (Name: LOForEach Schema: c1#14:int)
> | |
> | (Name: LOGenerate[false] Schema: c1#14:int)ColumnPrune:InputUids=[14]ColumnPrune:OutputUids=[14]
> | | |
> | | (Name: Cast Type: int Uid: 14)
> | | |
> | | |---c1:(Name: Project Type: bytearray Uid: 14 Input: 0 Column: (*))
> | |
> | |---(Name: LOInnerLoad[0] Schema: c1#14:bytearray)
> |
> |---TEST_DATA: (Name: LOLoad Schema: c1#14:bytearray)RequiredFields:null
> {code}
> Runs fine by turning off PushUpFilter ("-t PushUpFilter").
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)