You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Cheolsoo Park (JIRA)" <ji...@apache.org> on 2012/10/15 23:33:03 UTC
[jira] [Created] (PIG-2978) TestLoadStoreFuncLifeCycle fails with
hadoop-2.0.x
Cheolsoo Park created PIG-2978:
----------------------------------
Summary: TestLoadStoreFuncLifeCycle fails with hadoop-2.0.x
Key: PIG-2978
URL: https://issues.apache.org/jira/browse/PIG-2978
Project: Pig
Issue Type: Sub-task
Reporter: Cheolsoo Park
To reproduce, please run:
{code}
ant clean test -Dtestcase=TestLoadStoreFuncLifeCycle -Dhadoopversion=23
{code}
This fails with the following error:
{code}
Error during parsing. Job in state DEFINE instead of RUNNING
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Job in state DEFINE instead of RUNNING
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1607)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1546)
at org.apache.pig.PigServer.registerQuery(PigServer.java:516)
at org.apache.pig.PigServer.registerQuery(PigServer.java:529)
at org.apache.pig.TestLoadStoreFuncLifeCycle.testLoadStoreFunc(TestLoadStoreFuncLifeCycle.java:332)
Caused by: Failed to parse: Job in state DEFINE instead of RUNNING
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:193)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1599)
Caused by: java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING
at org.apache.hadoop.mapreduce.Job.ensureState(Job.java:292)
at org.apache.hadoop.mapreduce.Job.toString(Job.java:456)
at java.lang.String.valueOf(String.java:2826)
at org.apache.pig.TestLoadStoreFuncLifeCycle.logCaller(TestLoadStoreFuncLifeCycle.java:270)
at org.apache.pig.TestLoadStoreFuncLifeCycle.access$000(TestLoadStoreFuncLifeCycle.java:41)
at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.logCaller(TestLoadStoreFuncLifeCycle.java:54)
at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.getSchema(TestLoadStoreFuncLifeCycle.java:115)
at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:174)
at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:88)
at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:839)
at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3236)
at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1315)
at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:799)
at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:517)
at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:392)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2978) TestLoadStoreFuncLifeCycle fails with
hadoop-2.0.x
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478709#comment-13478709 ]
Cheolsoo Park commented on PIG-2978:
------------------------------------
Here is the difference between hadoop-1.0.x and 2.0.x:
{code:title=hadoop-1.0.x}
Storer[3].<init>()
Storer[3].setStoreFuncUDFContextSignature(A_1-1)
Storer[3].setStoreLocation(bar, org.apache.hadoop.mapreduce.Job)
Storer[3].getOutputFormat()
Storer[3].setStoreLocation(bar, org.apache.hadoop.mapreduce.Job)
{code}
{code:title=hadoop-2.0.x}
Storer[3].<init>()
Storer[3].setStoreFuncUDFContextSignature(A_1-1)
Storer[3].setStoreLocation(bar, org.apache.hadoop.mapreduce.Job)
Storer[3].getOutputFormat()
Storer[3].setStoreLocation(bar, org.apache.hadoop.mapreduce.Job)
Storer[4].<init>()
Storer[4].setStoreFuncUDFContextSignature(A_1-1)
Storer[4].setStoreLocation(bar, org.apache.hadoop.mapreduce.Job)
Storer[4].getOutputFormat()
Storer[4].setStoreLocation(bar, org.apache.hadoop.mapreduce.Job)
{code}
For whatever reason, getStoreFunc is repeated with hadoop-2.0.x. The call stack of the extra 4th instantiation is below:
{code}
Storer[4].<init> called by
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:577)
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:232)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:85)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:67)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
{code}
> TestLoadStoreFuncLifeCycle fails with hadoop-2.0.x
> --------------------------------------------------
>
> Key: PIG-2978
> URL: https://issues.apache.org/jira/browse/PIG-2978
> Project: Pig
> Issue Type: Sub-task
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Fix For: 0.11
>
> Attachments: PIG-2978.patch
>
>
> To reproduce, please run:
> {code}
> ant clean test -Dtestcase=TestLoadStoreFuncLifeCycle -Dhadoopversion=23
> {code}
> This fails with the following error:
> {code}
> Error during parsing. Job in state DEFINE instead of RUNNING
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Job in state DEFINE instead of RUNNING
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1607)
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1546)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:516)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:529)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.testLoadStoreFunc(TestLoadStoreFuncLifeCycle.java:332)
> Caused by: Failed to parse: Job in state DEFINE instead of RUNNING
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:193)
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1599)
> Caused by: java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING
> at org.apache.hadoop.mapreduce.Job.ensureState(Job.java:292)
> at org.apache.hadoop.mapreduce.Job.toString(Job.java:456)
> at java.lang.String.valueOf(String.java:2826)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.logCaller(TestLoadStoreFuncLifeCycle.java:270)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.access$000(TestLoadStoreFuncLifeCycle.java:41)
> at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.logCaller(TestLoadStoreFuncLifeCycle.java:54)
> at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.getSchema(TestLoadStoreFuncLifeCycle.java:115)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:174)
> at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:88)
> at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:839)
> at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3236)
> at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1315)
> at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:799)
> at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:517)
> at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:392)
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2978) TestLoadStoreFuncLifeCycle fails with
hadoop-2.0.x
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-2978:
-------------------------------
Status: Open (was: Patch Available)
Thanks Julien for your comment. I will take a further look at this.
> TestLoadStoreFuncLifeCycle fails with hadoop-2.0.x
> --------------------------------------------------
>
> Key: PIG-2978
> URL: https://issues.apache.org/jira/browse/PIG-2978
> Project: Pig
> Issue Type: Sub-task
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Fix For: 0.11
>
> Attachments: PIG-2978.patch
>
>
> To reproduce, please run:
> {code}
> ant clean test -Dtestcase=TestLoadStoreFuncLifeCycle -Dhadoopversion=23
> {code}
> This fails with the following error:
> {code}
> Error during parsing. Job in state DEFINE instead of RUNNING
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Job in state DEFINE instead of RUNNING
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1607)
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1546)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:516)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:529)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.testLoadStoreFunc(TestLoadStoreFuncLifeCycle.java:332)
> Caused by: Failed to parse: Job in state DEFINE instead of RUNNING
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:193)
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1599)
> Caused by: java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING
> at org.apache.hadoop.mapreduce.Job.ensureState(Job.java:292)
> at org.apache.hadoop.mapreduce.Job.toString(Job.java:456)
> at java.lang.String.valueOf(String.java:2826)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.logCaller(TestLoadStoreFuncLifeCycle.java:270)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.access$000(TestLoadStoreFuncLifeCycle.java:41)
> at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.logCaller(TestLoadStoreFuncLifeCycle.java:54)
> at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.getSchema(TestLoadStoreFuncLifeCycle.java:115)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:174)
> at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:88)
> at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:839)
> at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3236)
> at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1315)
> at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:799)
> at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:517)
> at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:392)
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2978) TestLoadStoreFuncLifeCycle fails with
hadoop-2.0.x
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-2978:
-------------------------------
Status: Patch Available (was: Open)
> TestLoadStoreFuncLifeCycle fails with hadoop-2.0.x
> --------------------------------------------------
>
> Key: PIG-2978
> URL: https://issues.apache.org/jira/browse/PIG-2978
> Project: Pig
> Issue Type: Sub-task
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Fix For: 0.11
>
> Attachments: PIG-2978-2.patch, PIG-2978.patch
>
>
> To reproduce, please run:
> {code}
> ant clean test -Dtestcase=TestLoadStoreFuncLifeCycle -Dhadoopversion=23
> {code}
> This fails with the following error:
> {code}
> Error during parsing. Job in state DEFINE instead of RUNNING
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Job in state DEFINE instead of RUNNING
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1607)
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1546)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:516)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:529)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.testLoadStoreFunc(TestLoadStoreFuncLifeCycle.java:332)
> Caused by: Failed to parse: Job in state DEFINE instead of RUNNING
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:193)
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1599)
> Caused by: java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING
> at org.apache.hadoop.mapreduce.Job.ensureState(Job.java:292)
> at org.apache.hadoop.mapreduce.Job.toString(Job.java:456)
> at java.lang.String.valueOf(String.java:2826)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.logCaller(TestLoadStoreFuncLifeCycle.java:270)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.access$000(TestLoadStoreFuncLifeCycle.java:41)
> at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.logCaller(TestLoadStoreFuncLifeCycle.java:54)
> at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.getSchema(TestLoadStoreFuncLifeCycle.java:115)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:174)
> at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:88)
> at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:839)
> at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3236)
> at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1315)
> at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:799)
> at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:517)
> at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:392)
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2978) TestLoadStoreFuncLifeCycle fails with
hadoop-2.0.x
Posted by "Julien Le Dem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500586#comment-13500586 ]
Julien Le Dem commented on PIG-2978:
------------------------------------
Thanks Cheolsoo for investigating this.
Yes the goal of this test is to avoid this kind of extra instance to creep in unnoticed.
It would be great to know why Pig gets a new POStore at that point.
> TestLoadStoreFuncLifeCycle fails with hadoop-2.0.x
> --------------------------------------------------
>
> Key: PIG-2978
> URL: https://issues.apache.org/jira/browse/PIG-2978
> Project: Pig
> Issue Type: Sub-task
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Fix For: 0.11
>
> Attachments: PIG-2978.patch
>
>
> To reproduce, please run:
> {code}
> ant clean test -Dtestcase=TestLoadStoreFuncLifeCycle -Dhadoopversion=23
> {code}
> This fails with the following error:
> {code}
> Error during parsing. Job in state DEFINE instead of RUNNING
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Job in state DEFINE instead of RUNNING
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1607)
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1546)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:516)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:529)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.testLoadStoreFunc(TestLoadStoreFuncLifeCycle.java:332)
> Caused by: Failed to parse: Job in state DEFINE instead of RUNNING
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:193)
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1599)
> Caused by: java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING
> at org.apache.hadoop.mapreduce.Job.ensureState(Job.java:292)
> at org.apache.hadoop.mapreduce.Job.toString(Job.java:456)
> at java.lang.String.valueOf(String.java:2826)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.logCaller(TestLoadStoreFuncLifeCycle.java:270)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.access$000(TestLoadStoreFuncLifeCycle.java:41)
> at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.logCaller(TestLoadStoreFuncLifeCycle.java:54)
> at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.getSchema(TestLoadStoreFuncLifeCycle.java:115)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:174)
> at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:88)
> at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:839)
> at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3236)
> at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1315)
> at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:799)
> at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:517)
> at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:392)
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2978) TestLoadStoreFuncLifeCycle fails with
hadoop-2.0.x
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-2978:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to 0.11/trunk.
Thanks Rohini for clarifying the difference in Hadoop 2.0.x.
> TestLoadStoreFuncLifeCycle fails with hadoop-2.0.x
> --------------------------------------------------
>
> Key: PIG-2978
> URL: https://issues.apache.org/jira/browse/PIG-2978
> Project: Pig
> Issue Type: Sub-task
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Fix For: 0.11
>
> Attachments: PIG-2978-2.patch, PIG-2978.patch
>
>
> To reproduce, please run:
> {code}
> ant clean test -Dtestcase=TestLoadStoreFuncLifeCycle -Dhadoopversion=23
> {code}
> This fails with the following error:
> {code}
> Error during parsing. Job in state DEFINE instead of RUNNING
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Job in state DEFINE instead of RUNNING
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1607)
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1546)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:516)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:529)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.testLoadStoreFunc(TestLoadStoreFuncLifeCycle.java:332)
> Caused by: Failed to parse: Job in state DEFINE instead of RUNNING
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:193)
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1599)
> Caused by: java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING
> at org.apache.hadoop.mapreduce.Job.ensureState(Job.java:292)
> at org.apache.hadoop.mapreduce.Job.toString(Job.java:456)
> at java.lang.String.valueOf(String.java:2826)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.logCaller(TestLoadStoreFuncLifeCycle.java:270)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.access$000(TestLoadStoreFuncLifeCycle.java:41)
> at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.logCaller(TestLoadStoreFuncLifeCycle.java:54)
> at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.getSchema(TestLoadStoreFuncLifeCycle.java:115)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:174)
> at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:88)
> at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:839)
> at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3236)
> at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1315)
> at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:799)
> at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:517)
> at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:392)
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2978) TestLoadStoreFuncLifeCycle fails with
hadoop-2.0.x
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-2978:
-------------------------------
Attachment: PIG-2978.patch
The error is thrown because the Hadoop-2.0.x Job class overrides the toString() method, and it checks whether the JobState is running or not.
{code}
@Override
public String toString() {
ensureState(JobState.RUNNING);
...
{code}
In hadoop-1.0.x, String.valueOf(<Job object>) is printed as something like this:
{code}
org.apache.hadoop.mapreduce.Job@7ee49dcd
{code}
So I replaced String.valueOf(<Job object>) with Job.class.getName().
With this change, the test now runs with hadoop-2.0.x but fails because the no. of store func instances is 5, whereas it is 4 with hadoop-1.0.x. So I replaced the assertion as follows:
{code}
Storer.count <= (Util.isHadoop2_0() ? 5 : 4)
{code}
It would be good if we could identify why it is different between hadoop-1.0.x and hadoop-2.0.x.
> TestLoadStoreFuncLifeCycle fails with hadoop-2.0.x
> --------------------------------------------------
>
> Key: PIG-2978
> URL: https://issues.apache.org/jira/browse/PIG-2978
> Project: Pig
> Issue Type: Sub-task
> Reporter: Cheolsoo Park
> Fix For: 0.11
>
> Attachments: PIG-2978.patch
>
>
> To reproduce, please run:
> {code}
> ant clean test -Dtestcase=TestLoadStoreFuncLifeCycle -Dhadoopversion=23
> {code}
> This fails with the following error:
> {code}
> Error during parsing. Job in state DEFINE instead of RUNNING
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Job in state DEFINE instead of RUNNING
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1607)
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1546)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:516)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:529)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.testLoadStoreFunc(TestLoadStoreFuncLifeCycle.java:332)
> Caused by: Failed to parse: Job in state DEFINE instead of RUNNING
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:193)
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1599)
> Caused by: java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING
> at org.apache.hadoop.mapreduce.Job.ensureState(Job.java:292)
> at org.apache.hadoop.mapreduce.Job.toString(Job.java:456)
> at java.lang.String.valueOf(String.java:2826)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.logCaller(TestLoadStoreFuncLifeCycle.java:270)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.access$000(TestLoadStoreFuncLifeCycle.java:41)
> at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.logCaller(TestLoadStoreFuncLifeCycle.java:54)
> at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.getSchema(TestLoadStoreFuncLifeCycle.java:115)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:174)
> at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:88)
> at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:839)
> at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3236)
> at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1315)
> at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:799)
> at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:517)
> at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:392)
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2978) TestLoadStoreFuncLifeCycle fails with
hadoop-2.0.x
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-2978:
-------------------------------
Attachment: PIG-2978-2.patch
Incorporated Rohini's comments in the RB.
- Changed Job.class.getName() to getJobName()
- Added comments regarding the difference between hadoop 1.0.x and 2.0.x in terms of the number of StoreFunc instances.
> TestLoadStoreFuncLifeCycle fails with hadoop-2.0.x
> --------------------------------------------------
>
> Key: PIG-2978
> URL: https://issues.apache.org/jira/browse/PIG-2978
> Project: Pig
> Issue Type: Sub-task
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Fix For: 0.11
>
> Attachments: PIG-2978-2.patch, PIG-2978.patch
>
>
> To reproduce, please run:
> {code}
> ant clean test -Dtestcase=TestLoadStoreFuncLifeCycle -Dhadoopversion=23
> {code}
> This fails with the following error:
> {code}
> Error during parsing. Job in state DEFINE instead of RUNNING
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Job in state DEFINE instead of RUNNING
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1607)
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1546)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:516)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:529)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.testLoadStoreFunc(TestLoadStoreFuncLifeCycle.java:332)
> Caused by: Failed to parse: Job in state DEFINE instead of RUNNING
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:193)
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1599)
> Caused by: java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING
> at org.apache.hadoop.mapreduce.Job.ensureState(Job.java:292)
> at org.apache.hadoop.mapreduce.Job.toString(Job.java:456)
> at java.lang.String.valueOf(String.java:2826)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.logCaller(TestLoadStoreFuncLifeCycle.java:270)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.access$000(TestLoadStoreFuncLifeCycle.java:41)
> at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.logCaller(TestLoadStoreFuncLifeCycle.java:54)
> at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.getSchema(TestLoadStoreFuncLifeCycle.java:115)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:174)
> at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:88)
> at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:839)
> at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3236)
> at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1315)
> at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:799)
> at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:517)
> at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:392)
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2978) TestLoadStoreFuncLifeCycle fails with
hadoop-2.0.x
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-2978:
-------------------------------
Assignee: Cheolsoo Park
Status: Patch Available (was: Open)
> TestLoadStoreFuncLifeCycle fails with hadoop-2.0.x
> --------------------------------------------------
>
> Key: PIG-2978
> URL: https://issues.apache.org/jira/browse/PIG-2978
> Project: Pig
> Issue Type: Sub-task
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Fix For: 0.11
>
> Attachments: PIG-2978.patch
>
>
> To reproduce, please run:
> {code}
> ant clean test -Dtestcase=TestLoadStoreFuncLifeCycle -Dhadoopversion=23
> {code}
> This fails with the following error:
> {code}
> Error during parsing. Job in state DEFINE instead of RUNNING
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Job in state DEFINE instead of RUNNING
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1607)
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1546)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:516)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:529)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.testLoadStoreFunc(TestLoadStoreFuncLifeCycle.java:332)
> Caused by: Failed to parse: Job in state DEFINE instead of RUNNING
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:193)
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1599)
> Caused by: java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING
> at org.apache.hadoop.mapreduce.Job.ensureState(Job.java:292)
> at org.apache.hadoop.mapreduce.Job.toString(Job.java:456)
> at java.lang.String.valueOf(String.java:2826)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.logCaller(TestLoadStoreFuncLifeCycle.java:270)
> at org.apache.pig.TestLoadStoreFuncLifeCycle.access$000(TestLoadStoreFuncLifeCycle.java:41)
> at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.logCaller(TestLoadStoreFuncLifeCycle.java:54)
> at org.apache.pig.TestLoadStoreFuncLifeCycle$InstrumentedStorage.getSchema(TestLoadStoreFuncLifeCycle.java:115)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:174)
> at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:88)
> at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:839)
> at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3236)
> at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1315)
> at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:799)
> at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:517)
> at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:392)
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira