You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Vivek Padmanabhan (JIRA)" <ji...@apache.org> on 2011/06/21 09:49:47 UTC
[jira] [Created] (PIG-2135) Pig 0.9 ignoring Multiple filter
conditions joined with AND/OR
Pig 0.9 ignoring Multiple filter conditions joined with AND/OR
---------------------------------------------------------------
Key: PIG-2135
URL: https://issues.apache.org/jira/browse/PIG-2135
Project: Pig
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Vivek Padmanabhan
Priority: Critical
Fix For: 0.9.0
When I have multiple filter statements joined by AND/OR , except for the first condition all other conditions are ignored.
For example in the below script the second condition (org.udfs.Func09('e',w) == 1) is ignored ;
a = load 'sample_input' using PigStorage(',') as (q:chararray,w:chararray);
b = filter a by org.udfs.Func09('f1',q) == 1 AND org.udfs.Func09('e',w) == 1 ;
dump b;
Output from the script
(f1,a)
(f1,e) --> this record should have been filtered by the second condition
Input for the script;
f1,a
f2,b
f3,c
f1,e
f2,f
f5,e
The explain of the alias b shows that the second condition is not included in the plan itself.
The above statements works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2135) Pig 0.9 ignoring Multiple filter
conditions joined with AND/OR
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053377#comment-13053377 ]
Thejas M Nair commented on PIG-2135:
------------------------------------
Pasting the above expression plan again, this time with proper formatting.
{code}
Both expressions are also there in explain plain -
#--------------------------------------------------
# Map Reduce Plan
#--------------------------------------------------
MapReduce node scope-43
Map Plan
b: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-42
|
|---b: Filter[bag] - scope-30
| |
| And[boolean] - scope-41
| |
| |---Equal To[boolean] - scope-35
| | |
| | |---POUserFunc(org.udfs.Func09)[int] - scope-33
| | | |
| | | |---Constant(f1) - scope-31
| | | |
| | | |---Project[chararray][0] - scope-32
| | |
| | |---Constant(1) - scope-34
| |
| |---Equal To[boolean] - scope-40
| |
| |---POUserFunc(org.udfs.Func09)[int] - scope-38
| | |
| | |---Constant(e) - scope-36
| | |
| | |---Project[chararray][1] - scope-37
| |
| |---Constant(1) - scope-39
|
|---a: New For Each(false,false)[bag] - scope-29
| |
| Cast[chararray] - scope-24
| |
| |---Project[bytearray][0] - scope-23
| |
| Cast[chararray] - scope-27
| |
| |---Project[bytearray][1] - scope-26
|
|---a: Load(hdfs://cobaltblue-nn1.blue.ygrid.yahoo.com/user/tejas/sample_input:PigStorage(',')) - scope-22--------
Global sort: false
----------------
{code}
> Pig 0.9 ignoring Multiple filter conditions joined with AND/OR
> ---------------------------------------------------------------
>
> Key: PIG-2135
> URL: https://issues.apache.org/jira/browse/PIG-2135
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Thejas M Nair
> Priority: Critical
> Fix For: 0.9.0
>
>
> When I have multiple filter statements joined by AND/OR , except for the first condition all other conditions are ignored.
> For example in the below script the second condition (org.udfs.Func09('e',w) == 1) is ignored ;
> a = load 'sample_input' using PigStorage(',') as (q:chararray,w:chararray);
> b = filter a by org.udfs.Func09('f1',q) == 1 AND org.udfs.Func09('e',w) == 1 ;
> dump b;
> Output from the script
> (f1,a)
> (f1,e) --> this record should have been filtered by the second condition
> Input for the script;
> f1,a
> f2,b
> f3,c
> f1,e
> f2,f
> f5,e
> The explain of the alias b shows that the second condition is not included in the plan itself.
> The above statements works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (PIG-2135) Pig 0.9 ignoring Multiple filter
conditions joined with AND/OR
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair reassigned PIG-2135:
----------------------------------
Assignee: Thejas M Nair
> Pig 0.9 ignoring Multiple filter conditions joined with AND/OR
> ---------------------------------------------------------------
>
> Key: PIG-2135
> URL: https://issues.apache.org/jira/browse/PIG-2135
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Thejas M Nair
> Priority: Critical
> Fix For: 0.9.0
>
>
> When I have multiple filter statements joined by AND/OR , except for the first condition all other conditions are ignored.
> For example in the below script the second condition (org.udfs.Func09('e',w) == 1) is ignored ;
> a = load 'sample_input' using PigStorage(',') as (q:chararray,w:chararray);
> b = filter a by org.udfs.Func09('f1',q) == 1 AND org.udfs.Func09('e',w) == 1 ;
> dump b;
> Output from the script
> (f1,a)
> (f1,e) --> this record should have been filtered by the second condition
> Input for the script;
> f1,a
> f2,b
> f3,c
> f1,e
> f2,f
> f5,e
> The explain of the alias b shows that the second condition is not included in the plan itself.
> The above statements works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2135) Pig 0.9 ignoring Multiple filter
conditions joined with AND/OR
Posted by "Vivek Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053380#comment-13053380 ]
Vivek Padmanabhan commented on PIG-2135:
----------------------------------------
Hi Thejas,
I was using an older version of Pig 0.9. This issue is not present in the latest code. Sorry about that.
> Pig 0.9 ignoring Multiple filter conditions joined with AND/OR
> ---------------------------------------------------------------
>
> Key: PIG-2135
> URL: https://issues.apache.org/jira/browse/PIG-2135
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Thejas M Nair
> Priority: Critical
> Fix For: 0.9.0
>
>
> When I have multiple filter statements joined by AND/OR , except for the first condition all other conditions are ignored.
> For example in the below script the second condition (org.udfs.Func09('e',w) == 1) is ignored ;
> a = load 'sample_input' using PigStorage(',') as (q:chararray,w:chararray);
> b = filter a by org.udfs.Func09('f1',q) == 1 AND org.udfs.Func09('e',w) == 1 ;
> dump b;
> Output from the script
> (f1,a)
> (f1,e) --> this record should have been filtered by the second condition
> Input for the script;
> f1,a
> f2,b
> f3,c
> f1,e
> f2,f
> f5,e
> The explain of the alias b shows that the second condition is not included in the plan itself.
> The above statements works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2135) Pig 0.9 ignoring Multiple filter
conditions joined with AND/OR
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053374#comment-13053374 ]
Thejas M Nair commented on PIG-2135:
------------------------------------
Vivek,
I am unable to reproduce this issue with 0.9, can you please check if the example in jira description is correct ?
grunt> register filudfs.jar
grunt> a = load 'sample_input' using PigStorage(',') as (q:chararray,w:chararray);
grunt> b = filter a by org.udfs.Func09('f1',q) == 1 AND org.udfs.Func09('e',w) == 1 ;
grunt> dump b;
(f1,e)
grunt>
Both expressions are also there in explain plain -
#--------------------------------------------------
# Map Reduce Plan
#--------------------------------------------------
MapReduce node scope-43
Map Plan
b: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-42
|
|---b: Filter[bag] - scope-30
| |
| And[boolean] - scope-41
| |
| |---Equal To[boolean] - scope-35
| | |
| | |---POUserFunc(org.udfs.Func09)[int] - scope-33
| | | |
| | | |---Constant(f1) - scope-31
| | | |
| | | |---Project[chararray][0] - scope-32
| | |
| | |---Constant(1) - scope-34
| |
| |---Equal To[boolean] - scope-40
| |
| |---POUserFunc(org.udfs.Func09)[int] - scope-38
| | |
| | |---Constant(e) - scope-36
| | |
| | |---Project[chararray][1] - scope-37
| |
| |---Constant(1) - scope-39
|
|---a: New For Each(false,false)[bag] - scope-29
| |
| Cast[chararray] - scope-24
| |
| |---Project[bytearray][0] - scope-23
| |
| Cast[chararray] - scope-27
| |
| |---Project[bytearray][1] - scope-26
|
|---a: Load(hdfs://cobaltblue-nn1.blue.ygrid.yahoo.com/user/tejas/sample_input:PigStorage(',')) - scope-22--------
Global sort: false
----------------
> Pig 0.9 ignoring Multiple filter conditions joined with AND/OR
> ---------------------------------------------------------------
>
> Key: PIG-2135
> URL: https://issues.apache.org/jira/browse/PIG-2135
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Thejas M Nair
> Priority: Critical
> Fix For: 0.9.0
>
>
> When I have multiple filter statements joined by AND/OR , except for the first condition all other conditions are ignored.
> For example in the below script the second condition (org.udfs.Func09('e',w) == 1) is ignored ;
> a = load 'sample_input' using PigStorage(',') as (q:chararray,w:chararray);
> b = filter a by org.udfs.Func09('f1',q) == 1 AND org.udfs.Func09('e',w) == 1 ;
> dump b;
> Output from the script
> (f1,a)
> (f1,e) --> this record should have been filtered by the second condition
> Input for the script;
> f1,a
> f2,b
> f3,c
> f1,e
> f2,f
> f5,e
> The explain of the alias b shows that the second condition is not included in the plan itself.
> The above statements works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2135) Pig 0.9 ignoring Multiple filter
conditions joined with AND/OR
Posted by "Vivek Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052414#comment-13052414 ]
Vivek Padmanabhan commented on PIG-2135:
----------------------------------------
UDF Used in the above example;
{code}
public class Func09 extends EvalFunc<Integer> {
@Override
public Integer exec(Tuple input) throws IOException {
String field1 = (String)input.get(0);
String field2 = (String)input.get(1);
return field1.equals(field2) ? 1 : 0;
}
}
{code}
> Pig 0.9 ignoring Multiple filter conditions joined with AND/OR
> ---------------------------------------------------------------
>
> Key: PIG-2135
> URL: https://issues.apache.org/jira/browse/PIG-2135
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Priority: Critical
> Fix For: 0.9.0
>
>
> When I have multiple filter statements joined by AND/OR , except for the first condition all other conditions are ignored.
> For example in the below script the second condition (org.udfs.Func09('e',w) == 1) is ignored ;
> a = load 'sample_input' using PigStorage(',') as (q:chararray,w:chararray);
> b = filter a by org.udfs.Func09('f1',q) == 1 AND org.udfs.Func09('e',w) == 1 ;
> dump b;
> Output from the script
> (f1,a)
> (f1,e) --> this record should have been filtered by the second condition
> Input for the script;
> f1,a
> f2,b
> f3,c
> f1,e
> f2,f
> f5,e
> The explain of the alias b shows that the second condition is not included in the plan itself.
> The above statements works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-2135) Pig 0.9 ignoring Multiple filter
conditions joined with AND/OR
Posted by "Vivek Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vivek Padmanabhan resolved PIG-2135.
------------------------------------
Resolution: Invalid
> Pig 0.9 ignoring Multiple filter conditions joined with AND/OR
> ---------------------------------------------------------------
>
> Key: PIG-2135
> URL: https://issues.apache.org/jira/browse/PIG-2135
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Thejas M Nair
> Priority: Critical
> Fix For: 0.9.0
>
>
> When I have multiple filter statements joined by AND/OR , except for the first condition all other conditions are ignored.
> For example in the below script the second condition (org.udfs.Func09('e',w) == 1) is ignored ;
> a = load 'sample_input' using PigStorage(',') as (q:chararray,w:chararray);
> b = filter a by org.udfs.Func09('f1',q) == 1 AND org.udfs.Func09('e',w) == 1 ;
> dump b;
> Output from the script
> (f1,a)
> (f1,e) --> this record should have been filtered by the second condition
> Input for the script;
> f1,a
> f2,b
> f3,c
> f1,e
> f2,f
> f5,e
> The explain of the alias b shows that the second condition is not included in the plan itself.
> The above statements works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira