You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Julien Le Dem (JIRA)" <ji...@apache.org> on 2012/04/24 00:08:33 UTC
[jira] [Created] (PIG-2659) add source location of the aliases in
the physical plan
Julien Le Dem created PIG-2659:
----------------------------------
Summary: add source location of the aliases in the physical plan
Key: PIG-2659
URL: https://issues.apache.org/jira/browse/PIG-2659
Project: Pig
Issue Type: Improvement
Components: impl
Reporter: Julien Le Dem
Assignee: Julien Le Dem
The goal is to provide better information about what is actually running in a job.
In particular when alias names are being reused.
For example with the following script:
{code}
A = LOAD 'foo' using PigStorage();
B = GROUP A BY $0;
A = FOREACH B GENERATE COUNT(A);
STORE A INTO 'bar';
{code}
The job conf will contain the following information
{code}
pig.alias.location: M: A[1,4],A[3,4],B[2,4] C: A[3,4],B[2,4] R: A[3,4]
{code}
A caveat is that the Logical Plan Optimizer throws away the original information when merging Logical Operators.
this is already the case today with pig.alias
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2659) add source location of the aliases in
the physical plan
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270903#comment-13270903 ]
Julien Le Dem commented on PIG-2659:
------------------------------------
Hi Daniel,
Yes my original motivation was to use this in the visualizer.
Note that I made the location transient for now because some tests are comparing serialized physical plans to a saved binary versions, which is very brittle and not precise enough. I.E.: it fails whatever you changed to the physical plan serialized representation.
explanation of the data
M: A[1,4],A[3,4],B[2,4] C: A[3,4],B[2,4] R: A[3,4]
M: Mapper plan contains the following aliases
C: Combiner plan contains the following aliases
R: reduce plane contains the following aliases
each item is:
alias[line,offset]
This could also be split in 3 different properties:
pig.alias.location.mapper: A[1,4],A[3,4],B[2,4]
pig.alias.location.combiner: A[3,4],B[2,4]
pig.alias.location.reducer: A[3,4]
Yes, we should also make sure merged operators info is maintained.
I will take care of the white spaces.
There is a test here:
test/org/apache/pig/newplan/logical/relational/TestLocationInPhysicalPlan.java
> add source location of the aliases in the physical plan
> -------------------------------------------------------
>
> Key: PIG-2659
> URL: https://issues.apache.org/jira/browse/PIG-2659
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Julien Le Dem
> Assignee: Julien Le Dem
> Attachments: PIG-2659.patch
>
>
> The goal is to provide better information about what is actually running in a job.
> In particular when alias names are being reused.
> For example with the following script:
> {code}
> A = LOAD 'foo' using PigStorage();
> B = GROUP A BY $0;
> A = FOREACH B GENERATE COUNT(A);
> STORE A INTO 'bar';
> {code}
> The job conf will contain the following information
> {code}
> pig.alias.location: M: A[1,4],A[3,4],B[2,4] C: A[3,4],B[2,4] R: A[3,4]
> {code}
> A caveat is that the Logical Plan Optimizer throws away the original information when merging Logical Operators.
> this is already the case today with pig.alias
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2659) add source location of the aliases in
the physical plan
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Le Dem updated PIG-2659:
-------------------------------
Attachment: PIG-2659_d.patch
PIG-2659_d.patch
adds logging as in PIG-2688
> add source location of the aliases in the physical plan
> -------------------------------------------------------
>
> Key: PIG-2659
> URL: https://issues.apache.org/jira/browse/PIG-2659
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Julien Le Dem
> Assignee: Julien Le Dem
> Attachments: PIG-2659.patch, PIG-2659_a.patch, PIG-2659_b.patch, PIG-2659_c.patch, PIG-2659_d.patch
>
>
> The goal is to provide better information about what is actually running in a job.
> In particular when alias names are being reused.
> For example with the following script:
> {code}
> A = LOAD 'foo' using PigStorage();
> B = GROUP A BY $0;
> A = FOREACH B GENERATE COUNT(A);
> STORE A INTO 'bar';
> {code}
> The job conf will contain the following information
> {code}
> pig.alias.location: M: A[1,4],A[3,4],B[2,4] C: A[3,4],B[2,4] R: A[3,4]
> {code}
> A caveat is that the Logical Plan Optimizer throws away the original information when merging Logical Operators.
> this is already the case today with pig.alias
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2659) add source location of the aliases in
the physical plan
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Le Dem updated PIG-2659:
-------------------------------
Attachment: PIG-2659_c.patch
PIG-2659_c.patch
makes the field not transient anymore so that it will be accessible in the backend. this is necessary as PigStats deserializes the plan from the job conf.
TestMRCompiler's golden files had to be updated accordingly when it should not have.
I opened a JIRA for this (PIG-2698)
> add source location of the aliases in the physical plan
> -------------------------------------------------------
>
> Key: PIG-2659
> URL: https://issues.apache.org/jira/browse/PIG-2659
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Julien Le Dem
> Assignee: Julien Le Dem
> Attachments: PIG-2659.patch, PIG-2659_a.patch, PIG-2659_b.patch, PIG-2659_c.patch
>
>
> The goal is to provide better information about what is actually running in a job.
> In particular when alias names are being reused.
> For example with the following script:
> {code}
> A = LOAD 'foo' using PigStorage();
> B = GROUP A BY $0;
> A = FOREACH B GENERATE COUNT(A);
> STORE A INTO 'bar';
> {code}
> The job conf will contain the following information
> {code}
> pig.alias.location: M: A[1,4],A[3,4],B[2,4] C: A[3,4],B[2,4] R: A[3,4]
> {code}
> A caveat is that the Logical Plan Optimizer throws away the original information when merging Logical Operators.
> this is already the case today with pig.alias
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2659) add source location of the aliases in
the physical plan
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270689#comment-13270689 ]
Daniel Dai commented on PIG-2659:
---------------------------------
That's awesome, with this, we can do:
1. Include line number for most backend exceptions
2. Include line number in explain
3. Include line number in stats
4. Use line number to generate job name
5. Make use of location in visualizer (PIG-2659)
Can you elaborate what "M: A[1,4],A[3,4],B[2,4] C: A[3,4],B[2,4] R: A[3,4]" means?
If we merge operators, we shall annotate how this operator come from, this can be done later.
Also be sure to remove white space changes when you checkin (svn diff -x --ignore-all-space).
Can you also add some tests?
> add source location of the aliases in the physical plan
> -------------------------------------------------------
>
> Key: PIG-2659
> URL: https://issues.apache.org/jira/browse/PIG-2659
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Julien Le Dem
> Assignee: Julien Le Dem
> Attachments: PIG-2659.patch
>
>
> The goal is to provide better information about what is actually running in a job.
> In particular when alias names are being reused.
> For example with the following script:
> {code}
> A = LOAD 'foo' using PigStorage();
> B = GROUP A BY $0;
> A = FOREACH B GENERATE COUNT(A);
> STORE A INTO 'bar';
> {code}
> The job conf will contain the following information
> {code}
> pig.alias.location: M: A[1,4],A[3,4],B[2,4] C: A[3,4],B[2,4] R: A[3,4]
> {code}
> A caveat is that the Logical Plan Optimizer throws away the original information when merging Logical Operators.
> this is already the case today with pig.alias
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2659) add source location of the aliases in
the physical plan
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Le Dem updated PIG-2659:
-------------------------------
Attachment: PIG-2659_a.patch
PIG-2659_a.patch is removing the unnecessary whitespace changes
> add source location of the aliases in the physical plan
> -------------------------------------------------------
>
> Key: PIG-2659
> URL: https://issues.apache.org/jira/browse/PIG-2659
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Julien Le Dem
> Assignee: Julien Le Dem
> Attachments: PIG-2659.patch, PIG-2659_a.patch
>
>
> The goal is to provide better information about what is actually running in a job.
> In particular when alias names are being reused.
> For example with the following script:
> {code}
> A = LOAD 'foo' using PigStorage();
> B = GROUP A BY $0;
> A = FOREACH B GENERATE COUNT(A);
> STORE A INTO 'bar';
> {code}
> The job conf will contain the following information
> {code}
> pig.alias.location: M: A[1,4],A[3,4],B[2,4] C: A[3,4],B[2,4] R: A[3,4]
> {code}
> A caveat is that the Logical Plan Optimizer throws away the original information when merging Logical Operators.
> this is already the case today with pig.alias
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-2659) add source location of the aliases in
the physical plan
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Le Dem resolved PIG-2659.
--------------------------------
Resolution: Fixed
Fix Version/s: 0.11
> add source location of the aliases in the physical plan
> -------------------------------------------------------
>
> Key: PIG-2659
> URL: https://issues.apache.org/jira/browse/PIG-2659
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Julien Le Dem
> Assignee: Julien Le Dem
> Fix For: 0.11
>
> Attachments: PIG-2659.patch, PIG-2659_a.patch, PIG-2659_b.patch, PIG-2659_c.patch, PIG-2659_d.patch
>
>
> The goal is to provide better information about what is actually running in a job.
> In particular when alias names are being reused.
> For example with the following script:
> {code}
> A = LOAD 'foo' using PigStorage();
> B = GROUP A BY $0;
> A = FOREACH B GENERATE COUNT(A);
> STORE A INTO 'bar';
> {code}
> The job conf will contain the following information
> {code}
> pig.alias.location: M: A[1,4],A[3,4],B[2,4] C: A[3,4],B[2,4] R: A[3,4]
> {code}
> A caveat is that the Logical Plan Optimizer throws away the original information when merging Logical Operators.
> this is already the case today with pig.alias
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2659) add source location of the aliases in
the physical plan
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Le Dem updated PIG-2659:
-------------------------------
Attachment: PIG-2659_b.patch
PIG-2659_b.patch rebased patch
> add source location of the aliases in the physical plan
> -------------------------------------------------------
>
> Key: PIG-2659
> URL: https://issues.apache.org/jira/browse/PIG-2659
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Julien Le Dem
> Assignee: Julien Le Dem
> Attachments: PIG-2659.patch, PIG-2659_a.patch, PIG-2659_b.patch
>
>
> The goal is to provide better information about what is actually running in a job.
> In particular when alias names are being reused.
> For example with the following script:
> {code}
> A = LOAD 'foo' using PigStorage();
> B = GROUP A BY $0;
> A = FOREACH B GENERATE COUNT(A);
> STORE A INTO 'bar';
> {code}
> The job conf will contain the following information
> {code}
> pig.alias.location: M: A[1,4],A[3,4],B[2,4] C: A[3,4],B[2,4] R: A[3,4]
> {code}
> A caveat is that the Logical Plan Optimizer throws away the original information when merging Logical Operators.
> this is already the case today with pig.alias
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2659) add source location of the aliases in
the physical plan
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270624#comment-13270624 ]
Bill Graham commented on PIG-2659:
----------------------------------
+1
Looks good to me. There were a lot of whitespace changes in this patch FYI.
> add source location of the aliases in the physical plan
> -------------------------------------------------------
>
> Key: PIG-2659
> URL: https://issues.apache.org/jira/browse/PIG-2659
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Julien Le Dem
> Assignee: Julien Le Dem
> Attachments: PIG-2659.patch
>
>
> The goal is to provide better information about what is actually running in a job.
> In particular when alias names are being reused.
> For example with the following script:
> {code}
> A = LOAD 'foo' using PigStorage();
> B = GROUP A BY $0;
> A = FOREACH B GENERATE COUNT(A);
> STORE A INTO 'bar';
> {code}
> The job conf will contain the following information
> {code}
> pig.alias.location: M: A[1,4],A[3,4],B[2,4] C: A[3,4],B[2,4] R: A[3,4]
> {code}
> A caveat is that the Logical Plan Optimizer throws away the original information when merging Logical Operators.
> this is already the case today with pig.alias
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2659) add source location of the aliases in
the physical plan
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270978#comment-13270978 ]
Daniel Dai commented on PIG-2659:
---------------------------------
Great, +1.
> add source location of the aliases in the physical plan
> -------------------------------------------------------
>
> Key: PIG-2659
> URL: https://issues.apache.org/jira/browse/PIG-2659
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Julien Le Dem
> Assignee: Julien Le Dem
> Attachments: PIG-2659.patch
>
>
> The goal is to provide better information about what is actually running in a job.
> In particular when alias names are being reused.
> For example with the following script:
> {code}
> A = LOAD 'foo' using PigStorage();
> B = GROUP A BY $0;
> A = FOREACH B GENERATE COUNT(A);
> STORE A INTO 'bar';
> {code}
> The job conf will contain the following information
> {code}
> pig.alias.location: M: A[1,4],A[3,4],B[2,4] C: A[3,4],B[2,4] R: A[3,4]
> {code}
> A caveat is that the Logical Plan Optimizer throws away the original information when merging Logical Operators.
> this is already the case today with pig.alias
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2659) add source location of the aliases in
the physical plan
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Le Dem updated PIG-2659:
-------------------------------
Attachment: PIG-2659.patch
PIG-2659.patch implements this
> add source location of the aliases in the physical plan
> -------------------------------------------------------
>
> Key: PIG-2659
> URL: https://issues.apache.org/jira/browse/PIG-2659
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Julien Le Dem
> Assignee: Julien Le Dem
> Attachments: PIG-2659.patch
>
>
> The goal is to provide better information about what is actually running in a job.
> In particular when alias names are being reused.
> For example with the following script:
> {code}
> A = LOAD 'foo' using PigStorage();
> B = GROUP A BY $0;
> A = FOREACH B GENERATE COUNT(A);
> STORE A INTO 'bar';
> {code}
> The job conf will contain the following information
> {code}
> pig.alias.location: M: A[1,4],A[3,4],B[2,4] C: A[3,4],B[2,4] R: A[3,4]
> {code}
> A caveat is that the Logical Plan Optimizer throws away the original information when merging Logical Operators.
> this is already the case today with pig.alias
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira