You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Olga Natkovich (JIRA)" <ji...@apache.org> on 2007/11/02 02:00:50 UTC
[jira] Created: (PIG-7) Optimize execution of algebraic functions
Optimize execution of algebraic functions
-----------------------------------------
Key: PIG-7
URL: https://issues.apache.org/jira/browse/PIG-7
Project: Pig
Issue Type: Improvement
Reporter: Olga Natkovich
Assignee: Alan Gates
Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-7) Optimize execution of algebraic functions
Posted by "Utkarsh Srivastava (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12548021 ]
Utkarsh Srivastava commented on PIG-7:
--------------------------------------
Looks good. But as Alan said, we should rework this code when there is more time.
2 comments one major, and one minor, both in PigCombine.java
Major:
Lines 93.94: Don't add the indexed tuple directly to the bag. We had a nasty bug a while back regarding this. Convert it into a regular tuple before adding it. see lines 148,149 in PigMapReduce.java
Minor: It would be nice to clean up the comments from PigCombine. Also, there are some fragments that don't make sense given the restricted setting we are applying the cominer in.
For example,
for (int i = 0; i < inputCount; i++) { // XXX: shouldn't we only do this if INNER flag is set?
if (t.getBagField(1 + i).isEmpty())
return;
}
Since we are currently running for the case when inputCount == 1, the bag will never be empty. (If the bag is empty, that group would never have been created).
> Optimize execution of algebraic functions
> -----------------------------------------
>
> Key: PIG-7
> URL: https://issues.apache.org/jira/browse/PIG-7
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Alan Gates
> Attachments: combiner.patch, combiner2.patch, combiner3.patch
>
>
> Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
> Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-7) Optimize execution of algebraic functions
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-7:
-----------------------------
Patch Info: (was: [Patch Available])
cleared patch available flag since all patches have been committed
> Optimize execution of algebraic functions
> -----------------------------------------
>
> Key: PIG-7
> URL: https://issues.apache.org/jira/browse/PIG-7
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Alan Gates
> Attachments: combiner.patch, combiner2.patch, combiner3.patch, pig7.patch
>
>
> Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
> Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-7) Optimize execution of algebraic functions
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12549266 ]
Alan Gates commented on PIG-7:
------------------------------
Checked in fix for IndexedTuples issue. Managed to reproduce the problem and test the patch before I checked it in.
> Optimize execution of algebraic functions
> -----------------------------------------
>
> Key: PIG-7
> URL: https://issues.apache.org/jira/browse/PIG-7
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Alan Gates
> Attachments: combiner.patch, combiner2.patch, combiner3.patch, pig7.patch
>
>
> Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
> Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-7) Optimize execution of algebraic functions
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-7:
-------------------------
Attachment: combiner.patch
> Optimize execution of algebraic functions
> -----------------------------------------
>
> Key: PIG-7
> URL: https://issues.apache.org/jira/browse/PIG-7
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Alan Gates
> Attachments: combiner.patch
>
>
> Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
> Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Updated: (PIG-7) Optimize execution of algebraic functions
Posted by Chris Olston <ol...@yahoo-inc.com>.
Awesome!!
On Nov 29, 2007, at 12:25 PM, Alan Gates (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/PIG-7?
> page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Alan Gates updated PIG-7:
> -------------------------
>
> Patch Info: [Patch Available]
>
> Attaching patch that implements use of combiner for algebraic
> functions in limited situations. Algebraic is only applied when
> all functions to be evaluated in a given generate line are
> algebraic and when there is one and only one relation being grouped
> (ie it is not applied in cogroup situations).
>
> Initial, very simple, performance tests show a speed up of ~40%
> (13m -> 7.5m for 4G on 10 machines) with the following script:
> a = load '/user/pig/tests/data/perf/studenttab200M';
> b = group a by $0;
> c = foreach b generate group, COUNT($1), SUM($1.$2), AVG($1.$2), MIN
> ($1.$1), MAX($1.$2);
> store c into 'bla';
>
>> Optimize execution of algebraic functions
>> -----------------------------------------
>>
>> Key: PIG-7
>> URL: https://issues.apache.org/jira/browse/PIG-7
>> Project: Pig
>> Issue Type: Improvement
>> Components: impl
>> Reporter: Olga Natkovich
>> Assignee: Alan Gates
>> Attachments: combiner.patch
>>
>>
>> Algebraic are functions that can be computed incrementally like
>> count(X), SUM(X), etc. They can be computed effciently by doing
>> the first level computation using hadoop combiner. This can give a
>> significant (2-3x) speedup for many aggregation queries.
>> Several users asked us for this feature so it is pretty high
>> priority.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
--
Christopher Olston, Ph.D.
Sr. Research Scientist
Yahoo! Research
[jira] Updated: (PIG-7) Optimize execution of algebraic functions
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-7:
-------------------------
Patch Info: [Patch Available]
Attaching patch that implements use of combiner for algebraic functions in limited situations. Algebraic is only applied when all functions to be evaluated in a given generate line are algebraic and when there is one and only one relation being grouped (ie it is not applied in cogroup situations).
Initial, very simple, performance tests show a speed up of ~40% (13m -> 7.5m for 4G on 10 machines) with the following script:
a = load '/user/pig/tests/data/perf/studenttab200M';
b = group a by $0;
c = foreach b generate group, COUNT($1), SUM($1.$2), AVG($1.$2), MIN($1.$1), MAX($1.$2);
store c into 'bla';
> Optimize execution of algebraic functions
> -----------------------------------------
>
> Key: PIG-7
> URL: https://issues.apache.org/jira/browse/PIG-7
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Alan Gates
> Attachments: combiner.patch
>
>
> Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
> Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-7) Optimize execution of algebraic functions
Posted by "Utkarsh Srivastava (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12548020 ]
Utkarsh Srivastava commented on PIG-7:
--------------------------------------
Looks good. But as Alan said, we should rework this code when there is more time.
2 comments one major, and one minor, both in PigCombine.java
Major:
Lines 93.94: Don't add the indexed tuple directly to the bag. We had a nasty bug a while back regarding this. Convert it into a regular tuple before adding it. see lines 148,149 in PigMapReduce.java
Minor: It would be nice to clean up the comments from PigCombine. Also, there are some fragments that don't make sense given the restricted setting we are applying the cominer in.
For example,
for (int i = 0; i < inputCount; i++) { // XXX: shouldn't we only do this if INNER flag is set?
if (t.getBagField(1 + i).isEmpty())
return;
}
Since we are currently running for the case when inputCount == 1, the bag will never be empty. (If the bag is empty, that group would never have been created).
> Optimize execution of algebraic functions
> -----------------------------------------
>
> Key: PIG-7
> URL: https://issues.apache.org/jira/browse/PIG-7
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Alan Gates
> Attachments: combiner.patch, combiner2.patch, combiner3.patch
>
>
> Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
> Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-7) Optimize execution of algebraic functions
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12547218 ]
Alan Gates commented on PIG-7:
------------------------------
Utkarsh,
Thanks for the feedback, I'll address the first two issues.
On the third issue, I have a question. Can you give me an example of a pig script that will not be combinable in this case? If I do something like:
a = load '/user/pig/tests/data/singlefile/studenttab10k';
b = group a by $0;
c = foreach b generate group, COUNT($1), SUM($1.$2);
d = filter c by $1 > '10';
dump d;
That still makes use of the combiner.
> Optimize execution of algebraic functions
> -----------------------------------------
>
> Key: PIG-7
> URL: https://issues.apache.org/jira/browse/PIG-7
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Alan Gates
> Attachments: combiner.patch
>
>
> Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
> Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-7) Optimize execution of
algebraic functions
Posted by "Utkarsh Srivastava (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12547012 ]
utkarsh edited comment on PIG-7 at 11/29/07 10:32 PM:
-----------------------------------------------------------------
Unfortunately, this patch has problems. It can kick off the combiner even in situations where it is not applicable. Following is the test sequence I tried:
finishship-lm-corp-yahoo-com:~/Documents/workspace/Test Pig $ cat b
a 1
a 2
b 3
b 4
finishship-lm-corp-yahoo-com:~/Documents/workspace/Test Pig $ java -cp pig.jar org.apache.pig.Main -
Connecting to hadoop file system at: localhost:9000
Connecting to map-reduce job tracker at: localhost:9001
grunt> a = load 'file:b';
grunt> b = group a by $0;
grunt> c = foreach b generate group, a;
grunt> dump c;
As you will see below, the combiner is kicked off while it shouldn't be, and then the job fails.
----- MapReduce Job -----
Input: [/tmp/temp-1447320079/tmp-1892534978:org.apache.pig.builtin.PigStorage()]
Map: [[*]]
Group: [GENERATE {[PROJECT $0],[*]}]
Combine: GENERATE {[PROJECT $0],[PROJECT $1]}
Reduce: GENERATE {[PROJECT $0],[PROJECT $1]}
Output: /tmp/temp-1447320079/tmp840894904:org.apache.pig.builtin.BinStorage
Split: null
Map parallelism: -1
Reduce parallelism: -1
Job jar size = 476828
Pig progress = 0%
Pig progress = 50%
Error message from task (map) tip_200711292202_0003_m_000000
Error message from task (reduce) tip_200711292202_0003_r_000000 java.io.IOException: Unexpected data while reading tuple from binary file
at org.apache.pig.data.Tuple.readFields(Tuple.java:294)
at org.apache.pig.data.DataBag.read(DataBag.java:251)
at org.apache.pig.data.Tuple.readDatum(Tuple.java:322)
at org.apache.pig.data.Tuple.read(Tuple.java:308)
at org.apache.pig.data.Tuple.readFields(Tuple.java:295)
at org.apache.pig.data.IndexedTuple.readFields(IndexedTuple.java:52)
at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.getNext(ReduceTask.java:210)
at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.<init>(ReduceTask.java:160)
at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.<init>(ReduceTask.java:228)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:320)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)
java.io.IOException: Unexpected data while reading tuple from binary file
I think the problem is that ProjectSpec unconditionally returns true for amenableToCombiner() while in the above example, it is not amenable.
Another, much smaller problem is that in visitSortDistinct() method, the sortSpec can be null (if the operator is carrying out a distinct), and that throws a NullPointerException (EvalSpecVisitor.java:62)
Another problem (though not strictly required to be solved in the first version), is that the combiner is kicked off in very restricted situations.
The condition in MapreducePlanCompiler.java is
if (mro.toReduce == null && spec.amenableToCombiner() &&
spec instanceof GenerateSpec &&
mro.groupFuncs != null && mro.groupFuncs.size() == 1) {
But, in most cases, the users will follow up GENERATE of SUM, AVG etc. by filter, or some other foreach etc. In these cases spec will be an instance of CompositeEvalSpec with the first thing as a GenerateSpec, and the combiner won't fire. It will be as easy to replace by a more general condition
spec instanceof GenerateSpec || (spec instanceof CompositeEvalSpec && ((CompositeEvalSpec)spec).getSpecs().get(0) instanceof GenerateSpec
was (Author: utkarsh):
Unfortunately, this patch has problems. It can kick off the combiner even in situations where it is not applicable. Following is the test sequence I tried:
finishship-lm-corp-yahoo-com:~/Documents/workspace/Test Pig $ java -cp pig.jar org.apache.pig.Main -
Connecting to hadoop file system at: localhost:9000
Connecting to map-reduce job tracker at: localhost:9001
grunt> a = load 'file:b';
grunt> b = group a by $0;
grunt> c = foreach b generate group, a;
grunt> dump c;
As you will see below, the combiner is kicked off while it shouldn't be, and then the job fails.
----- MapReduce Job -----
Input: [/tmp/temp-1447320079/tmp-1892534978:org.apache.pig.builtin.PigStorage()]
Map: [[*]]
Group: [GENERATE {[PROJECT $0],[*]}]
Combine: GENERATE {[PROJECT $0],[PROJECT $1]}
Reduce: GENERATE {[PROJECT $0],[PROJECT $1]}
Output: /tmp/temp-1447320079/tmp840894904:org.apache.pig.builtin.BinStorage
Split: null
Map parallelism: -1
Reduce parallelism: -1
Job jar size = 476828
Pig progress = 0%
Pig progress = 50%
Error message from task (map) tip_200711292202_0003_m_000000
Error message from task (reduce) tip_200711292202_0003_r_000000 java.io.IOException: Unexpected data while reading tuple from binary file
at org.apache.pig.data.Tuple.readFields(Tuple.java:294)
at org.apache.pig.data.DataBag.read(DataBag.java:251)
at org.apache.pig.data.Tuple.readDatum(Tuple.java:322)
at org.apache.pig.data.Tuple.read(Tuple.java:308)
at org.apache.pig.data.Tuple.readFields(Tuple.java:295)
at org.apache.pig.data.IndexedTuple.readFields(IndexedTuple.java:52)
at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.getNext(ReduceTask.java:210)
at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.<init>(ReduceTask.java:160)
at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.<init>(ReduceTask.java:228)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:320)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)
java.io.IOException: Unexpected data while reading tuple from binary file
I think the problem is that ProjectSpec unconditionally returns true for amenableToCombiner() while in the above example, it is not amenable.
Another, much smaller problem is that in visitSortDistinct() method, the sortSpec can be null (if the operator is carrying out a distinct), and that throws a NullPointerException (EvalSpecVisitor.java:62)
Another problem (though not strictly required to be solved in the first version), is that the combiner is kicked off in very restricted situations.
The condition in MapreducePlanCompiler.java is
if (mro.toReduce == null && spec.amenableToCombiner() &&
spec instanceof GenerateSpec &&
mro.groupFuncs != null && mro.groupFuncs.size() == 1) {
But, in most cases, the users will follow up GENERATE of SUM, AVG etc. by filter, or some other foreach etc. In these cases spec will be an instance of CompositeEvalSpec with the first thing as a GenerateSpec, and the combiner won't fire. It will be as easy to replace by a more general condition
spec instanceof GenerateSpec || (spec instanceof CompositeEvalSpec && ((CompositeEvalSpec)spec).getSpecs().get(0) instanceof GenerateSpec
> Optimize execution of algebraic functions
> -----------------------------------------
>
> Key: PIG-7
> URL: https://issues.apache.org/jira/browse/PIG-7
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Alan Gates
> Attachments: combiner.patch
>
>
> Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
> Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-7) Optimize execution of algebraic functions
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12548028 ]
Alan Gates commented on PIG-7:
------------------------------
Checked in patch combiner3.patch. I'm leaving the bug open for a couple of reasons.
1) We need to address the issues Utkarsh identified this comment: https://issues.apache.org/jira/browse/PIG-7#action_12548021
2) We need to expand the cases in which the combiner is used. In particular, we need to deal with the case where the group projection is somewhere other than the 0th position of the projection, or where it is not projected at all. This should not be too difficult, but we don't have time to fix it now.
> Optimize execution of algebraic functions
> -----------------------------------------
>
> Key: PIG-7
> URL: https://issues.apache.org/jira/browse/PIG-7
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Alan Gates
> Attachments: combiner.patch, combiner2.patch, combiner3.patch
>
>
> Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
> Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-7) Optimize execution of algebraic functions
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12549245 ]
Olga Natkovich commented on PIG-7:
----------------------------------
+1
> Optimize execution of algebraic functions
> -----------------------------------------
>
> Key: PIG-7
> URL: https://issues.apache.org/jira/browse/PIG-7
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Alan Gates
> Attachments: combiner.patch, combiner2.patch, combiner3.patch, pig7.patch
>
>
> Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
> Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-7) Optimize execution of algebraic functions
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-7:
-------------------------
Attachment: pig7.patch
This patch should resolve the issue mentioned by Utkarsh that spilling IndexedTuples causes problems.
> Optimize execution of algebraic functions
> -----------------------------------------
>
> Key: PIG-7
> URL: https://issues.apache.org/jira/browse/PIG-7
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Alan Gates
> Attachments: combiner.patch, combiner2.patch, combiner3.patch, pig7.patch
>
>
> Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
> Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-7) Optimize execution of algebraic functions
Posted by "Utkarsh Srivastava (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12547602 ]
Utkarsh Srivastava commented on PIG-7:
--------------------------------------
Still having a similar problem. The sequence I tried is below. The combiner shouldn't have kicked in. And then the job fails.
I think the correct solution is for CombinerVisiter to know which Project specs are amenable to combiner and which not. Thus immediately after group, $0 is amenable, but $1, $2 etc. are not. Also, another thing that the patch doesn't do yet is examine the arguments to the function. I am worried that bad things might happen if its something like SUM(blah($1)) instead of simply SUM($1). For now we can just target the cases when its of the simplest form such as some algebraic function($1).
finishship-lm-corp-yahoo-com:~/Documents/workspace/Test Pig $ java -cp eeyore/:pig.jar org.apache.pig.Main -
grunt> a = load 'a';
grunt> b = group a by $0;
grunt> c = foreach b generate $0, $1, SUM($1.$1);
grunt> dump c;
----- MapReduce Job -----
Input: [/user/utkarsh/a:org.apache.pig.builtin.PigStorage()]
Map: [[*]]
Group: [GENERATE {[PROJECT $0],[*]}]
Combine: GENERATE {[PROJECT $0],[PROJECT $1],[org.apache.pig.builtin.SUM$Initial(GENERATE {[PROJECT $1]->[PROJECT $1]})]}
Reduce: GENERATE {[PROJECT $0],[PROJECT $1],[org.apache.pig.builtin.SUM$Final(GENERATE {[PROJECT $1]->[PROJECT $2]})]}
Output: /tmp/temp1297608287/tmp242183746:org.apache.pig.builtin.BinStorage
Split: null
Map parallelism: -1
Reduce parallelism: -1
Job jar size = 477719
Pig progress = 0%
Pig progress = 50%
Error message from task (map) tip_200711301728_0003_m_000000
Error message from task (reduce) tip_200711301728_0003_r_000000 java.io.IOException: Unexpected data while reading tuple from binary file
at org.apache.pig.data.Tuple.readFields(Tuple.java:294)
at org.apache.pig.data.DataBag.read(DataBag.java:251)
> Optimize execution of algebraic functions
> -----------------------------------------
>
> Key: PIG-7
> URL: https://issues.apache.org/jira/browse/PIG-7
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Alan Gates
> Attachments: combiner.patch, combiner2.patch
>
>
> Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
> Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-7) Optimize execution of algebraic functions
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-7:
-----------------------------
Component/s: impl
> Optimize execution of algebraic functions
> -----------------------------------------
>
> Key: PIG-7
> URL: https://issues.apache.org/jira/browse/PIG-7
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Alan Gates
>
> Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
> Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-7) Optimize execution of algebraic functions
Posted by "Utkarsh Srivastava (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12547012 ]
Utkarsh Srivastava commented on PIG-7:
--------------------------------------
Unfortunately, this patch has problems. It can kick off the combiner even in situations where it is not applicable. Following is the test sequence I tried:
finishship-lm-corp-yahoo-com:~/Documents/workspace/Test Pig $ java -cp pig.jar org.apache.pig.Main -
Connecting to hadoop file system at: localhost:9000
Connecting to map-reduce job tracker at: localhost:9001
grunt> a = load 'file:b';
grunt> b = group a by $0;
grunt> c = foreach b generate group, a;
grunt> dump c;
As you will see below, the combiner is kicked off while it shouldn't be, and then the job fails.
----- MapReduce Job -----
Input: [/tmp/temp-1447320079/tmp-1892534978:org.apache.pig.builtin.PigStorage()]
Map: [[*]]
Group: [GENERATE {[PROJECT $0],[*]}]
Combine: GENERATE {[PROJECT $0],[PROJECT $1]}
Reduce: GENERATE {[PROJECT $0],[PROJECT $1]}
Output: /tmp/temp-1447320079/tmp840894904:org.apache.pig.builtin.BinStorage
Split: null
Map parallelism: -1
Reduce parallelism: -1
Job jar size = 476828
Pig progress = 0%
Pig progress = 50%
Error message from task (map) tip_200711292202_0003_m_000000
Error message from task (reduce) tip_200711292202_0003_r_000000 java.io.IOException: Unexpected data while reading tuple from binary file
at org.apache.pig.data.Tuple.readFields(Tuple.java:294)
at org.apache.pig.data.DataBag.read(DataBag.java:251)
at org.apache.pig.data.Tuple.readDatum(Tuple.java:322)
at org.apache.pig.data.Tuple.read(Tuple.java:308)
at org.apache.pig.data.Tuple.readFields(Tuple.java:295)
at org.apache.pig.data.IndexedTuple.readFields(IndexedTuple.java:52)
at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.getNext(ReduceTask.java:210)
at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.<init>(ReduceTask.java:160)
at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.<init>(ReduceTask.java:228)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:320)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)
java.io.IOException: Unexpected data while reading tuple from binary file
I think the problem is that ProjectSpec unconditionally returns true for amenableToCombiner() while in the above example, it is not amenable.
Another, much smaller problem is that in visitSortDistinct() method, the sortSpec can be null (if the operator is carrying out a distinct), and that throws a NullPointerException (EvalSpecVisitor.java:62)
Another problem (though not strictly required to be solved in the first version), is that the combiner is kicked off in very restricted situations.
The condition in MapreducePlanCompiler.java is
if (mro.toReduce == null && spec.amenableToCombiner() &&
spec instanceof GenerateSpec &&
mro.groupFuncs != null && mro.groupFuncs.size() == 1) {
But, in most cases, the users will follow up GENERATE of SUM, AVG etc. by filter, or some other foreach etc. In these cases spec will be an instance of CompositeEvalSpec with the first thing as a GenerateSpec, and the combiner won't fire. It will be as easy to replace by a more general condition
spec instanceof GenerateSpec || (spec instanceof CompositeEvalSpec && ((CompositeEvalSpec)spec).getSpecs().get(0) instanceof GenerateSpec
> Optimize execution of algebraic functions
> -----------------------------------------
>
> Key: PIG-7
> URL: https://issues.apache.org/jira/browse/PIG-7
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Alan Gates
> Attachments: combiner.patch
>
>
> Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
> Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-7) Optimize execution of algebraic functions
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-7:
-------------------------
Attachment: combiner2.patch
Here is a second pass at the combiner patch. I fixed:
1) the possible null pointer exception in EvalSpecVisitor pointed out by Utkarsh in pt 2 of his previous comment.
2) I dealt with the overly lax choice of the combiner (pt 1 in Utkarsh's previous mail) by changing the way the way combinability is determined. I have removed amenableToCombiner from EvalSpec and it's children. There is now a visitor CombineDeterminer that extends EvalSpecVisitor and is a private class of MadreducePlanCompiler. This visitor will only decide to use the combiner if it sees at least one FuncEvalSpec that has an Algebraic function and it does not see any eval specs that cannot be combined. The previous version (using amenableToCombiner) only did the latter.
I have rerun the unit tests and all pass.
> Optimize execution of algebraic functions
> -----------------------------------------
>
> Key: PIG-7
> URL: https://issues.apache.org/jira/browse/PIG-7
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Alan Gates
> Attachments: combiner.patch, combiner2.patch
>
>
> Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
> Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-7) Optimize execution of algebraic functions
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-7:
-------------------------
Attachment: combiner3.patch
The patch combiner3.patch addresses Utkarsh's points that the previous code wasn't handling the case where there was a func(func()) in the projection. It also wasn't handling the case where the projection was anything other than: group, func(), [func()...]. Both of those are explicitly caught now.
One note is that the use of the combiner in this patch is fairly restrictive. The user has to have a projection with the group in the position 0. We should probably rework this so that the group can either be omitted or moved around. I don't have time to do this now, but it shouldn't be too much work and it will make using the code more flexible.
> Optimize execution of algebraic functions
> -----------------------------------------
>
> Key: PIG-7
> URL: https://issues.apache.org/jira/browse/PIG-7
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Alan Gates
> Attachments: combiner.patch, combiner2.patch, combiner3.patch
>
>
> Algebraic are functions that can be computed incrementally like count(X), SUM(X), etc. They can be computed effciently by doing the first level computation using hadoop combiner. This can give a significant (2-3x) speedup for many aggregation queries.
> Several users asked us for this feature so it is pretty high priority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.