You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Vivek Padmanabhan (Created) (JIRA)" <ji...@apache.org> on 2011/11/23 09:05:40 UTC
[jira] [Created] (PIG-2385) Store statements not getting processed
Store statements not getting processed
--------------------------------------
Key: PIG-2385
URL: https://issues.apache.org/jira/browse/PIG-2385
Project: Pig
Issue Type: Bug
Affects Versions: 0.9.1
Reporter: Vivek Padmanabhan
Priority: Critical
The actual script in which we got this issue is pretty big and complex. The script has total 4 STORE statements and one of the STORE statement is not getting executed.
The script executes 3 sets of jobs (excluding one STORE which is not getting executed) consisting of 10, 11 and 19 jobs.
The below script could be used to illustrate the issue but with Multiquery turned off;
{code}
A = LOAD 'input1' as (f1:chararray,f2:chararray,f3:chararray);
Z = group A all;
Z1 = foreach Z generate COUNT(A) as count;
B = foreach A generate f1,f2,f3,(100-Z1.count) as diff;
C = order B by diff;
STORE C INTO 'output/C_out';
D = DISTINCT C ;
store D into 'output/F_out';
{code}
For this script, if run with Multiquery turned off, the Store command for D is not getting executed.
I can see that the statements are getting parsed and LOStore created for D , but still, it is not getting executed.
The above script works fine with Pig 0.8.(This issue still exists in Trunk as well)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2385) Store statements not getting processed
Posted by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2385:
----------------------------
Fix Version/s: 0.11
0.10
> Store statements not getting processed
> --------------------------------------
>
> Key: PIG-2385
> URL: https://issues.apache.org/jira/browse/PIG-2385
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Vivek Padmanabhan
> Assignee: Daniel Dai
> Priority: Critical
> Fix For: 0.10, 0.9.2, 0.11
>
> Attachments: PIG-2385-0.patch, PIG-2385-1.patch
>
>
> The actual script in which we got this issue is pretty big and complex. The script has total 4 STORE statements and one of the STORE statement is not getting executed.
> The script executes 3 sets of jobs (excluding one STORE which is not getting executed) consisting of 10, 11 and 19 jobs.
> The below script could be used to illustrate the issue but with Multiquery turned off;
> {code}
> A = LOAD 'input1' as (f1:chararray,f2:chararray,f3:chararray);
> Z = group A all;
> Z1 = foreach Z generate COUNT(A) as count;
> B = foreach A generate f1,f2,f3,(100-Z1.count) as diff;
> C = order B by diff;
> STORE C INTO 'output/C_out';
> D = DISTINCT C ;
> store D into 'output/F_out';
> {code}
> For this script, if run with Multiquery turned off, the Store command for D is not getting executed.
> I can see that the statements are getting parsed and LOStore created for D , but still, it is not getting executed.
> The above script works fine with Pig 0.8.(This issue still exists in Trunk as well)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2385) Store statements not getting
processed
Posted by "Daniel Dai (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161096#comment-13161096 ]
Daniel Dai commented on PIG-2385:
---------------------------------
I only need to add test case, I will try to upload the patch today.
> Store statements not getting processed
> --------------------------------------
>
> Key: PIG-2385
> URL: https://issues.apache.org/jira/browse/PIG-2385
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Vivek Padmanabhan
> Assignee: Daniel Dai
> Priority: Critical
> Fix For: 0.9.2
>
> Attachments: PIG-2385-0.patch
>
>
> The actual script in which we got this issue is pretty big and complex. The script has total 4 STORE statements and one of the STORE statement is not getting executed.
> The script executes 3 sets of jobs (excluding one STORE which is not getting executed) consisting of 10, 11 and 19 jobs.
> The below script could be used to illustrate the issue but with Multiquery turned off;
> {code}
> A = LOAD 'input1' as (f1:chararray,f2:chararray,f3:chararray);
> Z = group A all;
> Z1 = foreach Z generate COUNT(A) as count;
> B = foreach A generate f1,f2,f3,(100-Z1.count) as diff;
> C = order B by diff;
> STORE C INTO 'output/C_out';
> D = DISTINCT C ;
> store D into 'output/F_out';
> {code}
> For this script, if run with Multiquery turned off, the Store command for D is not getting executed.
> I can see that the statements are getting parsed and LOStore created for D , but still, it is not getting executed.
> The above script works fine with Pig 0.8.(This issue still exists in Trunk as well)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2385) Store statements not getting processed
Posted by "Olga Natkovich (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-2385:
--------------------------------
Fix Version/s: 0.9.2
Assignee: Daniel Dai
> Store statements not getting processed
> --------------------------------------
>
> Key: PIG-2385
> URL: https://issues.apache.org/jira/browse/PIG-2385
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Vivek Padmanabhan
> Assignee: Daniel Dai
> Priority: Critical
> Fix For: 0.9.2
>
> Attachments: PIG-2385-0.patch
>
>
> The actual script in which we got this issue is pretty big and complex. The script has total 4 STORE statements and one of the STORE statement is not getting executed.
> The script executes 3 sets of jobs (excluding one STORE which is not getting executed) consisting of 10, 11 and 19 jobs.
> The below script could be used to illustrate the issue but with Multiquery turned off;
> {code}
> A = LOAD 'input1' as (f1:chararray,f2:chararray,f3:chararray);
> Z = group A all;
> Z1 = foreach Z generate COUNT(A) as count;
> B = foreach A generate f1,f2,f3,(100-Z1.count) as diff;
> C = order B by diff;
> STORE C INTO 'output/C_out';
> D = DISTINCT C ;
> store D into 'output/F_out';
> {code}
> For this script, if run with Multiquery turned off, the Store command for D is not getting executed.
> I can see that the statements are getting parsed and LOStore created for D , but still, it is not getting executed.
> The above script works fine with Pig 0.8.(This issue still exists in Trunk as well)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2385) Store statements not getting
processed
Posted by "Vivek Padmanabhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155748#comment-13155748 ]
Vivek Padmanabhan commented on PIG-2385:
----------------------------------------
I think that this issue is introduced as part of new parser changes in Pig 0.9.
As per my analysis, the Scalar ref used,"Z1.count" introduces another additional LOStore with a tmp location.
While processing for STORE D, it considers this extra LOStore also and thinks that it has processed all store statements.
Hence,the store operator for the alias 'D' is skipped (this is happening at PigServer.Graph.skipStores() ).
The above script, could be used to replicate the scenario without multiquery, if we use PigServer to register and execute the queries.
> Store statements not getting processed
> --------------------------------------
>
> Key: PIG-2385
> URL: https://issues.apache.org/jira/browse/PIG-2385
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Vivek Padmanabhan
> Priority: Critical
>
> The actual script in which we got this issue is pretty big and complex. The script has total 4 STORE statements and one of the STORE statement is not getting executed.
> The script executes 3 sets of jobs (excluding one STORE which is not getting executed) consisting of 10, 11 and 19 jobs.
> The below script could be used to illustrate the issue but with Multiquery turned off;
> {code}
> A = LOAD 'input1' as (f1:chararray,f2:chararray,f3:chararray);
> Z = group A all;
> Z1 = foreach Z generate COUNT(A) as count;
> B = foreach A generate f1,f2,f3,(100-Z1.count) as diff;
> C = order B by diff;
> STORE C INTO 'output/C_out';
> D = DISTINCT C ;
> store D into 'output/F_out';
> {code}
> For this script, if run with Multiquery turned off, the Store command for D is not getting executed.
> I can see that the statements are getting parsed and LOStore created for D , but still, it is not getting executed.
> The above script works fine with Pig 0.8.(This issue still exists in Trunk as well)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2385) Store statements not getting
processed
Posted by "Vivek Padmanabhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163420#comment-13163420 ]
Vivek Padmanabhan commented on PIG-2385:
----------------------------------------
Thanks for the patch Daniel. I verified the patch with the actual script and its working fine.
Just for the update, the actual script in which we saw this issue does not have an exec statement and multiquery is on.
> Store statements not getting processed
> --------------------------------------
>
> Key: PIG-2385
> URL: https://issues.apache.org/jira/browse/PIG-2385
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Vivek Padmanabhan
> Assignee: Daniel Dai
> Priority: Critical
> Fix For: 0.10, 0.9.2, 0.11
>
> Attachments: PIG-2385-0.patch, PIG-2385-1.patch
>
>
> The actual script in which we got this issue is pretty big and complex. The script has total 4 STORE statements and one of the STORE statement is not getting executed.
> The script executes 3 sets of jobs (excluding one STORE which is not getting executed) consisting of 10, 11 and 19 jobs.
> The below script could be used to illustrate the issue but with Multiquery turned off;
> {code}
> A = LOAD 'input1' as (f1:chararray,f2:chararray,f3:chararray);
> Z = group A all;
> Z1 = foreach Z generate COUNT(A) as count;
> B = foreach A generate f1,f2,f3,(100-Z1.count) as diff;
> C = order B by diff;
> STORE C INTO 'output/C_out';
> D = DISTINCT C ;
> store D into 'output/F_out';
> {code}
> For this script, if run with Multiquery turned off, the Store command for D is not getting executed.
> I can see that the statements are getting parsed and LOStore created for D , but still, it is not getting executed.
> The above script works fine with Pig 0.8.(This issue still exists in Trunk as well)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2385) Store statements not getting processed
Posted by "Olga Natkovich (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-2385:
--------------------------------
Daniel,
When do you think this can be checked in? We would like this on 0.9 branch in addition to 10 and trunk, thanks
> Store statements not getting processed
> --------------------------------------
>
> Key: PIG-2385
> URL: https://issues.apache.org/jira/browse/PIG-2385
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Vivek Padmanabhan
> Assignee: Daniel Dai
> Priority: Critical
> Fix For: 0.9.2
>
> Attachments: PIG-2385-0.patch
>
>
> The actual script in which we got this issue is pretty big and complex. The script has total 4 STORE statements and one of the STORE statement is not getting executed.
> The script executes 3 sets of jobs (excluding one STORE which is not getting executed) consisting of 10, 11 and 19 jobs.
> The below script could be used to illustrate the issue but with Multiquery turned off;
> {code}
> A = LOAD 'input1' as (f1:chararray,f2:chararray,f3:chararray);
> Z = group A all;
> Z1 = foreach Z generate COUNT(A) as count;
> B = foreach A generate f1,f2,f3,(100-Z1.count) as diff;
> C = order B by diff;
> STORE C INTO 'output/C_out';
> D = DISTINCT C ;
> store D into 'output/F_out';
> {code}
> For this script, if run with Multiquery turned off, the Store command for D is not getting executed.
> I can see that the statements are getting parsed and LOStore created for D , but still, it is not getting executed.
> The above script works fine with Pig 0.8.(This issue still exists in Trunk as well)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2385) Store statements not getting
processed
Posted by "Daniel Dai (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162943#comment-13162943 ]
Daniel Dai commented on PIG-2385:
---------------------------------
Hi, Olga, the conditions are:
1. Use scalar
2. Multiple stores in script
3. Muti-query is off, or the multi-query is on, but you have "exec" in the middle of script(the script before exec contains scalar definition)
When this happens, Pig does not even create empty directory.
> Store statements not getting processed
> --------------------------------------
>
> Key: PIG-2385
> URL: https://issues.apache.org/jira/browse/PIG-2385
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Vivek Padmanabhan
> Assignee: Daniel Dai
> Priority: Critical
> Fix For: 0.10, 0.9.2, 0.11
>
> Attachments: PIG-2385-0.patch, PIG-2385-1.patch
>
>
> The actual script in which we got this issue is pretty big and complex. The script has total 4 STORE statements and one of the STORE statement is not getting executed.
> The script executes 3 sets of jobs (excluding one STORE which is not getting executed) consisting of 10, 11 and 19 jobs.
> The below script could be used to illustrate the issue but with Multiquery turned off;
> {code}
> A = LOAD 'input1' as (f1:chararray,f2:chararray,f3:chararray);
> Z = group A all;
> Z1 = foreach Z generate COUNT(A) as count;
> B = foreach A generate f1,f2,f3,(100-Z1.count) as diff;
> C = order B by diff;
> STORE C INTO 'output/C_out';
> D = DISTINCT C ;
> store D into 'output/F_out';
> {code}
> For this script, if run with Multiquery turned off, the Store command for D is not getting executed.
> I can see that the statements are getting parsed and LOStore created for D , but still, it is not getting executed.
> The above script works fine with Pig 0.8.(This issue still exists in Trunk as well)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2385) Store statements not getting processed
Posted by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2385:
----------------------------
Attachment: PIG-2385-0.patch
Attach a draft patch.
> Store statements not getting processed
> --------------------------------------
>
> Key: PIG-2385
> URL: https://issues.apache.org/jira/browse/PIG-2385
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Vivek Padmanabhan
> Priority: Critical
> Attachments: PIG-2385-0.patch
>
>
> The actual script in which we got this issue is pretty big and complex. The script has total 4 STORE statements and one of the STORE statement is not getting executed.
> The script executes 3 sets of jobs (excluding one STORE which is not getting executed) consisting of 10, 11 and 19 jobs.
> The below script could be used to illustrate the issue but with Multiquery turned off;
> {code}
> A = LOAD 'input1' as (f1:chararray,f2:chararray,f3:chararray);
> Z = group A all;
> Z1 = foreach Z generate COUNT(A) as count;
> B = foreach A generate f1,f2,f3,(100-Z1.count) as diff;
> C = order B by diff;
> STORE C INTO 'output/C_out';
> D = DISTINCT C ;
> store D into 'output/F_out';
> {code}
> For this script, if run with Multiquery turned off, the Store command for D is not getting executed.
> I can see that the statements are getting parsed and LOStore created for D , but still, it is not getting executed.
> The above script works fine with Pig 0.8.(This issue still exists in Trunk as well)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2385) Store statements not getting
processed
Posted by "Thejas M Nair (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161924#comment-13161924 ]
Thejas M Nair commented on PIG-2385:
------------------------------------
+1
> Store statements not getting processed
> --------------------------------------
>
> Key: PIG-2385
> URL: https://issues.apache.org/jira/browse/PIG-2385
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Vivek Padmanabhan
> Assignee: Daniel Dai
> Priority: Critical
> Fix For: 0.9.2
>
> Attachments: PIG-2385-0.patch, PIG-2385-1.patch
>
>
> The actual script in which we got this issue is pretty big and complex. The script has total 4 STORE statements and one of the STORE statement is not getting executed.
> The script executes 3 sets of jobs (excluding one STORE which is not getting executed) consisting of 10, 11 and 19 jobs.
> The below script could be used to illustrate the issue but with Multiquery turned off;
> {code}
> A = LOAD 'input1' as (f1:chararray,f2:chararray,f3:chararray);
> Z = group A all;
> Z1 = foreach Z generate COUNT(A) as count;
> B = foreach A generate f1,f2,f3,(100-Z1.count) as diff;
> C = order B by diff;
> STORE C INTO 'output/C_out';
> D = DISTINCT C ;
> store D into 'output/F_out';
> {code}
> For this script, if run with Multiquery turned off, the Store command for D is not getting executed.
> I can see that the statements are getting parsed and LOStore created for D , but still, it is not getting executed.
> The above script works fine with Pig 0.8.(This issue still exists in Trunk as well)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2385) Store statements not getting
processed
Posted by "Olga Natkovich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162933#comment-13162933 ]
Olga Natkovich commented on PIG-2385:
-------------------------------------
What are the conditions under which the issue observed?
Also, I assume not even an empty directory created when a store is missed, is this correct?
> Store statements not getting processed
> --------------------------------------
>
> Key: PIG-2385
> URL: https://issues.apache.org/jira/browse/PIG-2385
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Vivek Padmanabhan
> Assignee: Daniel Dai
> Priority: Critical
> Fix For: 0.10, 0.9.2, 0.11
>
> Attachments: PIG-2385-0.patch, PIG-2385-1.patch
>
>
> The actual script in which we got this issue is pretty big and complex. The script has total 4 STORE statements and one of the STORE statement is not getting executed.
> The script executes 3 sets of jobs (excluding one STORE which is not getting executed) consisting of 10, 11 and 19 jobs.
> The below script could be used to illustrate the issue but with Multiquery turned off;
> {code}
> A = LOAD 'input1' as (f1:chararray,f2:chararray,f3:chararray);
> Z = group A all;
> Z1 = foreach Z generate COUNT(A) as count;
> B = foreach A generate f1,f2,f3,(100-Z1.count) as diff;
> C = order B by diff;
> STORE C INTO 'output/C_out';
> D = DISTINCT C ;
> store D into 'output/F_out';
> {code}
> For this script, if run with Multiquery turned off, the Store command for D is not getting executed.
> I can see that the statements are getting parsed and LOStore created for D , but still, it is not getting executed.
> The above script works fine with Pig 0.8.(This issue still exists in Trunk as well)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2385) Store statements not getting
processed
Posted by "Daniel Dai (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158265#comment-13158265 ]
Daniel Dai commented on PIG-2385:
---------------------------------
We use processedStores to track how many stores we've already processed. However, we counting different things. When we count how many stores we processed, we use the plan after postProcess. When we skip stores in the logical plan, we use the plan before postProcess. The difference is the former count the scalar LOStore added by postProcess, the later not. We should make the count on the same plan.
> Store statements not getting processed
> --------------------------------------
>
> Key: PIG-2385
> URL: https://issues.apache.org/jira/browse/PIG-2385
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Vivek Padmanabhan
> Priority: Critical
>
> The actual script in which we got this issue is pretty big and complex. The script has total 4 STORE statements and one of the STORE statement is not getting executed.
> The script executes 3 sets of jobs (excluding one STORE which is not getting executed) consisting of 10, 11 and 19 jobs.
> The below script could be used to illustrate the issue but with Multiquery turned off;
> {code}
> A = LOAD 'input1' as (f1:chararray,f2:chararray,f3:chararray);
> Z = group A all;
> Z1 = foreach Z generate COUNT(A) as count;
> B = foreach A generate f1,f2,f3,(100-Z1.count) as diff;
> C = order B by diff;
> STORE C INTO 'output/C_out';
> D = DISTINCT C ;
> store D into 'output/F_out';
> {code}
> For this script, if run with Multiquery turned off, the Store command for D is not getting executed.
> I can see that the statements are getting parsed and LOStore created for D , but still, it is not getting executed.
> The above script works fine with Pig 0.8.(This issue still exists in Trunk as well)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-2385) Store statements not getting processed
Posted by "Daniel Dai (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai resolved PIG-2385.
-----------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Unit tests pass. test-patch:
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] -1 release audit. The applied patch generated 463 release audit warnings (more than the trunk's current 456 warnings).
No new file added, ignore release audit warning.
Patch committed to trunk/0.10/0.11
> Store statements not getting processed
> --------------------------------------
>
> Key: PIG-2385
> URL: https://issues.apache.org/jira/browse/PIG-2385
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Vivek Padmanabhan
> Assignee: Daniel Dai
> Priority: Critical
> Fix For: 0.9.2
>
> Attachments: PIG-2385-0.patch, PIG-2385-1.patch
>
>
> The actual script in which we got this issue is pretty big and complex. The script has total 4 STORE statements and one of the STORE statement is not getting executed.
> The script executes 3 sets of jobs (excluding one STORE which is not getting executed) consisting of 10, 11 and 19 jobs.
> The below script could be used to illustrate the issue but with Multiquery turned off;
> {code}
> A = LOAD 'input1' as (f1:chararray,f2:chararray,f3:chararray);
> Z = group A all;
> Z1 = foreach Z generate COUNT(A) as count;
> B = foreach A generate f1,f2,f3,(100-Z1.count) as diff;
> C = order B by diff;
> STORE C INTO 'output/C_out';
> D = DISTINCT C ;
> store D into 'output/F_out';
> {code}
> For this script, if run with Multiquery turned off, the Store command for D is not getting executed.
> I can see that the statements are getting parsed and LOStore created for D , but still, it is not getting executed.
> The above script works fine with Pig 0.8.(This issue still exists in Trunk as well)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2385) Store statements not getting processed
Posted by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2385:
----------------------------
Attachment: PIG-2385-1.patch
> Store statements not getting processed
> --------------------------------------
>
> Key: PIG-2385
> URL: https://issues.apache.org/jira/browse/PIG-2385
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: Vivek Padmanabhan
> Assignee: Daniel Dai
> Priority: Critical
> Fix For: 0.9.2
>
> Attachments: PIG-2385-0.patch, PIG-2385-1.patch
>
>
> The actual script in which we got this issue is pretty big and complex. The script has total 4 STORE statements and one of the STORE statement is not getting executed.
> The script executes 3 sets of jobs (excluding one STORE which is not getting executed) consisting of 10, 11 and 19 jobs.
> The below script could be used to illustrate the issue but with Multiquery turned off;
> {code}
> A = LOAD 'input1' as (f1:chararray,f2:chararray,f3:chararray);
> Z = group A all;
> Z1 = foreach Z generate COUNT(A) as count;
> B = foreach A generate f1,f2,f3,(100-Z1.count) as diff;
> C = order B by diff;
> STORE C INTO 'output/C_out';
> D = DISTINCT C ;
> store D into 'output/F_out';
> {code}
> For this script, if run with Multiquery turned off, the Store command for D is not getting executed.
> I can see that the statements are getting parsed and LOStore created for D , but still, it is not getting executed.
> The above script works fine with Pig 0.8.(This issue still exists in Trunk as well)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira