You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Mitesh Singh Jat (Created) (JIRA)" <ji...@apache.org> on 2012/01/24 05:49:40 UTC
[jira] [Created] (PIG-2489) Input Path Globbing{} not working with
PigStorageSchema or PigStorage('\t', '-schema');
Input Path Globbing{} not working with PigStorageSchema or PigStorage('\t', '-schema');
---------------------------------------------------------------------------------------
Key: PIG-2489
URL: https://issues.apache.org/jira/browse/PIG-2489
Project: Pig
Issue Type: Bug
Affects Versions: 0.9.1, 0.9.0, 0.10
Reporter: Mitesh Singh Jat
{code:title=test.pig}
-- For Pig 0.9
--A = LOAD 'input/PigStorageSchema/Temp{1,2}/pss*' USING org.apache.pig.piggybank.storage.PigStorageSchema();
-- For Pig 0.10
A = LOAD 'input/PigStorageSchema/Temp{1,2}/pss*' USING PigStorage('\t', '-schema');
DESCRIBE A;
DUMP A
{code}
Schema file _input/PigStorageSchema/Temp{1,2}.pig_schema_
{code}
{"fields":[{"name":"name","type":55,"schema":null,"description":"autogenerated from Pig Field Schema"},{"name":"val","type":10,"schema":null,"description":"autogenerated from Pig Field Schema"}],"version":0,"sortKeys":[],"sortKeyOrders":[]}
{code}
Header file _input/PigStorageSchema/Temp{1,2}/.pig_header_
{code}
name val
{code}
Sample input file _input/PigStorageSchema/Temp1/pss.in_
{code}
peter 1
samir 3
michael 4
peter 2
peter 4
samir 1
{code}
On running the above pig script _test.pig_ with pig 0.10, the following error is received.
{noformat}
012-01-24 04:07:42,210 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
{noformat}
Pig Stack Trace
{noformat}
Pig Stack Trace
---------------
ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias A
at org.apache.pig.PigServer.openIterator(PigServer.java:858)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:655)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:567)
at org.apache.pig.Main.main(Main.java:111)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias A
at org.apache.pig.PigServer.storeEx(PigServer.java:957)
at org.apache.pig.PigServer.store(PigServer.java:920)
at org.apache.pig.PigServer.openIterator(PigServer.java:833)
... 7 more
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2245:
<file test.pig, line 2, column 4> Cannot get schema from loadFunc org.apache.pig.builtin.PigStorage
at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:154)
at org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:109)
at org.apache.pig.newplan.logical.relational.LOStore.getSchema(LOStore.java:68)
at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.validate(SchemaAliasVisitor.java:60)
at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.visit(SchemaAliasVisitor.java:84)
at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
at org.apache.pig.PigServer$Graph.compile(PigServer.java:1618)
at org.apache.pig.PigServer$Graph.compile(PigServer.java:1612)
at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1335)
at org.apache.pig.PigServer.storeEx(PigServer.java:952)
... 9 more
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
at org.apache.pig.builtin.JsonMetadata.nullOrException(JsonMetadata.java:222)
at org.apache.pig.builtin.JsonMetadata.getSchema(JsonMetadata.java:191)
at org.apache.pig.builtin.PigStorage.getSchema(PigStorage.java:438)
at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:150)
{noformat}
Whereas PigStorageSchema() or PigStorage('\t', '-schema') works with *wildcard* *.
For example, following script works
{code:title=test2.pig}
A = LOAD 'input/PigStorageSchema/Temp*/pss*' USING PigStorage('\t', '-schema');
DESCRIBE A;
DUMP A;
{code}
As a workaround to make Temp{1,2} globbing work, the *,(comma)* separated multiple input paths (with no globbing)
can given as input.
{code:title=test2.pig}
A = LOAD 'input/PigStorageSchema/Temp1/pss*,input/PigStorageSchema/Temp2/pss*' USING PigStorage('\t', '-schema');
DESCRIBE A;
DUMP A;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-2489) Input Path Globbing{} not working with
PigStorageSchema or PigStorage('\t', '-schema');
Posted by "Daniel Dai (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai resolved PIG-2489.
-----------------------------
Resolution: Fixed
Fix Version/s: 0.11
0.10
Assignee: Daniel Dai
Hadoop Flags: Reviewed
Unit tests pass. test-patch:
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] -1 release audit. The applied patch generated 527 release audit warnings (more than the trunk's current 524 warnings).
javadoc warning is unrelated. No new file added, ignore release audit warning.
Patch committed to 0.10/trunk.
> Input Path Globbing{} not working with PigStorageSchema or PigStorage('\t', '-schema');
> ---------------------------------------------------------------------------------------
>
> Key: PIG-2489
> URL: https://issues.apache.org/jira/browse/PIG-2489
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.1, 0.10
> Reporter: Mitesh Singh Jat
> Assignee: Daniel Dai
> Fix For: 0.10, 0.11
>
> Attachments: PIG-2489-1.patch
>
>
> {code:title=test.pig}
> -- For Pig 0.9
> --A = LOAD 'input/PigStorageSchema/Temp{1,2}/pss*' USING org.apache.pig.piggybank.storage.PigStorageSchema();
> -- For Pig 0.10
> A = LOAD 'input/PigStorageSchema/Temp{1,2}/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A
> {code}
> Schema file _input/PigStorageSchema/Temp{1,2}.pig_schema_
> {code}
> {"fields":[{"name":"name","type":55,"schema":null,"description":"autogenerated from Pig Field Schema"},{"name":"val","type":10,"schema":null,"description":"autogenerated from Pig Field Schema"}],"version":0,"sortKeys":[],"sortKeyOrders":[]}
> {code}
> Header file _input/PigStorageSchema/Temp{1,2}/.pig_header_
> {code}
> name val
> {code}
> Sample input file _input/PigStorageSchema/Temp1/pss.in_
> {code}
> peter 1
> samir 3
> michael 4
> peter 2
> peter 4
> samir 1
> {code}
> On running the above pig script _test.pig_ with pig 0.10, the following error is received.
> {noformat}
> 012-01-24 04:07:42,210 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> {noformat}
> Pig Stack Trace
> {noformat}
> Pig Stack Trace
> ---------------
> ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias A
> at org.apache.pig.PigServer.openIterator(PigServer.java:858)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:655)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:567)
> at org.apache.pig.Main.main(Main.java:111)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias A
> at org.apache.pig.PigServer.storeEx(PigServer.java:957)
> at org.apache.pig.PigServer.store(PigServer.java:920)
> at org.apache.pig.PigServer.openIterator(PigServer.java:833)
> ... 7 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2245:
> <file test.pig, line 2, column 4> Cannot get schema from loadFunc org.apache.pig.builtin.PigStorage
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:154)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:109)
> at org.apache.pig.newplan.logical.relational.LOStore.getSchema(LOStore.java:68)
> at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.validate(SchemaAliasVisitor.java:60)
> at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.visit(SchemaAliasVisitor.java:84)
> at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1618)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1612)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1335)
> at org.apache.pig.PigServer.storeEx(PigServer.java:952)
> ... 9 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> at org.apache.pig.builtin.JsonMetadata.nullOrException(JsonMetadata.java:222)
> at org.apache.pig.builtin.JsonMetadata.getSchema(JsonMetadata.java:191)
> at org.apache.pig.builtin.PigStorage.getSchema(PigStorage.java:438)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:150)
> {noformat}
> Whereas PigStorageSchema() or PigStorage('\t', '-schema') works with *wildcard* *.
> For example, following script works
> {code:title=test2.pig}
> A = LOAD 'input/PigStorageSchema/Temp*/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A;
> {code}
> As a workaround to make Temp{1,2} globbing work, the *,(comma)* separated multiple input paths (with no globbing)
> can given as input.
> {code:title=test2.pig}
> A = LOAD 'input/PigStorageSchema/Temp1/pss*,input/PigStorageSchema/Temp2/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A;
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2489) Input Path Globbing{} not working
with PigStorageSchema or PigStorage('\t', '-schema');
Posted by "Vivek Padmanabhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192123#comment-13192123 ]
Vivek Padmanabhan commented on PIG-2489:
----------------------------------------
With the updated branch, for a script like below (i1,i2 are directories),
A = LOAD '2489/i{1,2}/' USING PigStorage('\t', '-schema');
DESCRIBE A;
DUMP A;
I am still getting the below error (even with patch from PIG-2209 );
"Caused by: java.net.URISyntaxException: Illegal character in path at index 65: hdfs://namnode:8020/user/pvivek/2489/i{1,2}"
Looking at JsonMetadata.findMetaFile(String path, String metaname, Configuration conf)
| storage = new HDataStorage(new URI(loc), ConfigurationUtil.toProperties(conf));
If I change this to
|storage = new HDataStorage((new org.apache.hadoop.fs.Path(loc)).toUri(),ConfigurationUtil.toProperties(conf));
The scripts are running fine for Pig 0.10.
Please help me in understanding how this case is handled in trunk.
Thanks
> Input Path Globbing{} not working with PigStorageSchema or PigStorage('\t', '-schema');
> ---------------------------------------------------------------------------------------
>
> Key: PIG-2489
> URL: https://issues.apache.org/jira/browse/PIG-2489
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.1, 0.10
> Reporter: Mitesh Singh Jat
>
> {code:title=test.pig}
> -- For Pig 0.9
> --A = LOAD 'input/PigStorageSchema/Temp{1,2}/pss*' USING org.apache.pig.piggybank.storage.PigStorageSchema();
> -- For Pig 0.10
> A = LOAD 'input/PigStorageSchema/Temp{1,2}/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A
> {code}
> Schema file _input/PigStorageSchema/Temp{1,2}.pig_schema_
> {code}
> {"fields":[{"name":"name","type":55,"schema":null,"description":"autogenerated from Pig Field Schema"},{"name":"val","type":10,"schema":null,"description":"autogenerated from Pig Field Schema"}],"version":0,"sortKeys":[],"sortKeyOrders":[]}
> {code}
> Header file _input/PigStorageSchema/Temp{1,2}/.pig_header_
> {code}
> name val
> {code}
> Sample input file _input/PigStorageSchema/Temp1/pss.in_
> {code}
> peter 1
> samir 3
> michael 4
> peter 2
> peter 4
> samir 1
> {code}
> On running the above pig script _test.pig_ with pig 0.10, the following error is received.
> {noformat}
> 012-01-24 04:07:42,210 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> {noformat}
> Pig Stack Trace
> {noformat}
> Pig Stack Trace
> ---------------
> ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias A
> at org.apache.pig.PigServer.openIterator(PigServer.java:858)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:655)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:567)
> at org.apache.pig.Main.main(Main.java:111)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias A
> at org.apache.pig.PigServer.storeEx(PigServer.java:957)
> at org.apache.pig.PigServer.store(PigServer.java:920)
> at org.apache.pig.PigServer.openIterator(PigServer.java:833)
> ... 7 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2245:
> <file test.pig, line 2, column 4> Cannot get schema from loadFunc org.apache.pig.builtin.PigStorage
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:154)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:109)
> at org.apache.pig.newplan.logical.relational.LOStore.getSchema(LOStore.java:68)
> at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.validate(SchemaAliasVisitor.java:60)
> at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.visit(SchemaAliasVisitor.java:84)
> at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1618)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1612)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1335)
> at org.apache.pig.PigServer.storeEx(PigServer.java:952)
> ... 9 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> at org.apache.pig.builtin.JsonMetadata.nullOrException(JsonMetadata.java:222)
> at org.apache.pig.builtin.JsonMetadata.getSchema(JsonMetadata.java:191)
> at org.apache.pig.builtin.PigStorage.getSchema(PigStorage.java:438)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:150)
> {noformat}
> Whereas PigStorageSchema() or PigStorage('\t', '-schema') works with *wildcard* *.
> For example, following script works
> {code:title=test2.pig}
> A = LOAD 'input/PigStorageSchema/Temp*/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A;
> {code}
> As a workaround to make Temp{1,2} globbing work, the *,(comma)* separated multiple input paths (with no globbing)
> can given as input.
> {code:title=test2.pig}
> A = LOAD 'input/PigStorageSchema/Temp1/pss*,input/PigStorageSchema/Temp2/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A;
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2489) Input Path Globbing{} not working with
PigStorageSchema or PigStorage('\t', '-schema');
Posted by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2489:
----------------------------
Attachment: PIG-2489-1.patch
> Input Path Globbing{} not working with PigStorageSchema or PigStorage('\t', '-schema');
> ---------------------------------------------------------------------------------------
>
> Key: PIG-2489
> URL: https://issues.apache.org/jira/browse/PIG-2489
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.1, 0.10
> Reporter: Mitesh Singh Jat
> Attachments: PIG-2489-1.patch
>
>
> {code:title=test.pig}
> -- For Pig 0.9
> --A = LOAD 'input/PigStorageSchema/Temp{1,2}/pss*' USING org.apache.pig.piggybank.storage.PigStorageSchema();
> -- For Pig 0.10
> A = LOAD 'input/PigStorageSchema/Temp{1,2}/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A
> {code}
> Schema file _input/PigStorageSchema/Temp{1,2}.pig_schema_
> {code}
> {"fields":[{"name":"name","type":55,"schema":null,"description":"autogenerated from Pig Field Schema"},{"name":"val","type":10,"schema":null,"description":"autogenerated from Pig Field Schema"}],"version":0,"sortKeys":[],"sortKeyOrders":[]}
> {code}
> Header file _input/PigStorageSchema/Temp{1,2}/.pig_header_
> {code}
> name val
> {code}
> Sample input file _input/PigStorageSchema/Temp1/pss.in_
> {code}
> peter 1
> samir 3
> michael 4
> peter 2
> peter 4
> samir 1
> {code}
> On running the above pig script _test.pig_ with pig 0.10, the following error is received.
> {noformat}
> 012-01-24 04:07:42,210 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> {noformat}
> Pig Stack Trace
> {noformat}
> Pig Stack Trace
> ---------------
> ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias A
> at org.apache.pig.PigServer.openIterator(PigServer.java:858)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:655)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:567)
> at org.apache.pig.Main.main(Main.java:111)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias A
> at org.apache.pig.PigServer.storeEx(PigServer.java:957)
> at org.apache.pig.PigServer.store(PigServer.java:920)
> at org.apache.pig.PigServer.openIterator(PigServer.java:833)
> ... 7 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2245:
> <file test.pig, line 2, column 4> Cannot get schema from loadFunc org.apache.pig.builtin.PigStorage
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:154)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:109)
> at org.apache.pig.newplan.logical.relational.LOStore.getSchema(LOStore.java:68)
> at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.validate(SchemaAliasVisitor.java:60)
> at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.visit(SchemaAliasVisitor.java:84)
> at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1618)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1612)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1335)
> at org.apache.pig.PigServer.storeEx(PigServer.java:952)
> ... 9 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> at org.apache.pig.builtin.JsonMetadata.nullOrException(JsonMetadata.java:222)
> at org.apache.pig.builtin.JsonMetadata.getSchema(JsonMetadata.java:191)
> at org.apache.pig.builtin.PigStorage.getSchema(PigStorage.java:438)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:150)
> {noformat}
> Whereas PigStorageSchema() or PigStorage('\t', '-schema') works with *wildcard* *.
> For example, following script works
> {code:title=test2.pig}
> A = LOAD 'input/PigStorageSchema/Temp*/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A;
> {code}
> As a workaround to make Temp{1,2} globbing work, the *,(comma)* separated multiple input paths (with no globbing)
> can given as input.
> {code:title=test2.pig}
> A = LOAD 'input/PigStorageSchema/Temp1/pss*,input/PigStorageSchema/Temp2/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A;
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2489) Input Path Globbing{} not working
with PigStorageSchema or PigStorage('\t', '-schema');
Posted by "Dmitriy V. Ryaboy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191890#comment-13191890 ]
Dmitriy V. Ryaboy commented on PIG-2489:
----------------------------------------
This is fixed in trunk.
> Input Path Globbing{} not working with PigStorageSchema or PigStorage('\t', '-schema');
> ---------------------------------------------------------------------------------------
>
> Key: PIG-2489
> URL: https://issues.apache.org/jira/browse/PIG-2489
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.1, 0.10
> Reporter: Mitesh Singh Jat
>
> {code:title=test.pig}
> -- For Pig 0.9
> --A = LOAD 'input/PigStorageSchema/Temp{1,2}/pss*' USING org.apache.pig.piggybank.storage.PigStorageSchema();
> -- For Pig 0.10
> A = LOAD 'input/PigStorageSchema/Temp{1,2}/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A
> {code}
> Schema file _input/PigStorageSchema/Temp{1,2}.pig_schema_
> {code}
> {"fields":[{"name":"name","type":55,"schema":null,"description":"autogenerated from Pig Field Schema"},{"name":"val","type":10,"schema":null,"description":"autogenerated from Pig Field Schema"}],"version":0,"sortKeys":[],"sortKeyOrders":[]}
> {code}
> Header file _input/PigStorageSchema/Temp{1,2}/.pig_header_
> {code}
> name val
> {code}
> Sample input file _input/PigStorageSchema/Temp1/pss.in_
> {code}
> peter 1
> samir 3
> michael 4
> peter 2
> peter 4
> samir 1
> {code}
> On running the above pig script _test.pig_ with pig 0.10, the following error is received.
> {noformat}
> 012-01-24 04:07:42,210 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> {noformat}
> Pig Stack Trace
> {noformat}
> Pig Stack Trace
> ---------------
> ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias A
> at org.apache.pig.PigServer.openIterator(PigServer.java:858)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:655)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:567)
> at org.apache.pig.Main.main(Main.java:111)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias A
> at org.apache.pig.PigServer.storeEx(PigServer.java:957)
> at org.apache.pig.PigServer.store(PigServer.java:920)
> at org.apache.pig.PigServer.openIterator(PigServer.java:833)
> ... 7 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2245:
> <file test.pig, line 2, column 4> Cannot get schema from loadFunc org.apache.pig.builtin.PigStorage
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:154)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:109)
> at org.apache.pig.newplan.logical.relational.LOStore.getSchema(LOStore.java:68)
> at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.validate(SchemaAliasVisitor.java:60)
> at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.visit(SchemaAliasVisitor.java:84)
> at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1618)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1612)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1335)
> at org.apache.pig.PigServer.storeEx(PigServer.java:952)
> ... 9 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> at org.apache.pig.builtin.JsonMetadata.nullOrException(JsonMetadata.java:222)
> at org.apache.pig.builtin.JsonMetadata.getSchema(JsonMetadata.java:191)
> at org.apache.pig.builtin.PigStorage.getSchema(PigStorage.java:438)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:150)
> {noformat}
> Whereas PigStorageSchema() or PigStorage('\t', '-schema') works with *wildcard* *.
> For example, following script works
> {code:title=test2.pig}
> A = LOAD 'input/PigStorageSchema/Temp*/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A;
> {code}
> As a workaround to make Temp{1,2} globbing work, the *,(comma)* separated multiple input paths (with no globbing)
> can given as input.
> {code:title=test2.pig}
> A = LOAD 'input/PigStorageSchema/Temp1/pss*,input/PigStorageSchema/Temp2/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A;
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2489) Input Path Globbing{} not working
with PigStorageSchema or PigStorage('\t', '-schema');
Posted by "Thejas M Nair (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202008#comment-13202008 ]
Thejas M Nair commented on PIG-2489:
------------------------------------
+1
> Input Path Globbing{} not working with PigStorageSchema or PigStorage('\t', '-schema');
> ---------------------------------------------------------------------------------------
>
> Key: PIG-2489
> URL: https://issues.apache.org/jira/browse/PIG-2489
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.1, 0.10
> Reporter: Mitesh Singh Jat
> Attachments: PIG-2489-1.patch
>
>
> {code:title=test.pig}
> -- For Pig 0.9
> --A = LOAD 'input/PigStorageSchema/Temp{1,2}/pss*' USING org.apache.pig.piggybank.storage.PigStorageSchema();
> -- For Pig 0.10
> A = LOAD 'input/PigStorageSchema/Temp{1,2}/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A
> {code}
> Schema file _input/PigStorageSchema/Temp{1,2}.pig_schema_
> {code}
> {"fields":[{"name":"name","type":55,"schema":null,"description":"autogenerated from Pig Field Schema"},{"name":"val","type":10,"schema":null,"description":"autogenerated from Pig Field Schema"}],"version":0,"sortKeys":[],"sortKeyOrders":[]}
> {code}
> Header file _input/PigStorageSchema/Temp{1,2}/.pig_header_
> {code}
> name val
> {code}
> Sample input file _input/PigStorageSchema/Temp1/pss.in_
> {code}
> peter 1
> samir 3
> michael 4
> peter 2
> peter 4
> samir 1
> {code}
> On running the above pig script _test.pig_ with pig 0.10, the following error is received.
> {noformat}
> 012-01-24 04:07:42,210 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> {noformat}
> Pig Stack Trace
> {noformat}
> Pig Stack Trace
> ---------------
> ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias A
> at org.apache.pig.PigServer.openIterator(PigServer.java:858)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:655)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:567)
> at org.apache.pig.Main.main(Main.java:111)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias A
> at org.apache.pig.PigServer.storeEx(PigServer.java:957)
> at org.apache.pig.PigServer.store(PigServer.java:920)
> at org.apache.pig.PigServer.openIterator(PigServer.java:833)
> ... 7 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2245:
> <file test.pig, line 2, column 4> Cannot get schema from loadFunc org.apache.pig.builtin.PigStorage
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:154)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:109)
> at org.apache.pig.newplan.logical.relational.LOStore.getSchema(LOStore.java:68)
> at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.validate(SchemaAliasVisitor.java:60)
> at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.visit(SchemaAliasVisitor.java:84)
> at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1618)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1612)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1335)
> at org.apache.pig.PigServer.storeEx(PigServer.java:952)
> ... 9 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> at org.apache.pig.builtin.JsonMetadata.nullOrException(JsonMetadata.java:222)
> at org.apache.pig.builtin.JsonMetadata.getSchema(JsonMetadata.java:191)
> at org.apache.pig.builtin.PigStorage.getSchema(PigStorage.java:438)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:150)
> {noformat}
> Whereas PigStorageSchema() or PigStorage('\t', '-schema') works with *wildcard* *.
> For example, following script works
> {code:title=test2.pig}
> A = LOAD 'input/PigStorageSchema/Temp*/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A;
> {code}
> As a workaround to make Temp{1,2} globbing work, the *,(comma)* separated multiple input paths (with no globbing)
> can given as input.
> {code:title=test2.pig}
> A = LOAD 'input/PigStorageSchema/Temp1/pss*,input/PigStorageSchema/Temp2/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A;
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2489) Input Path Globbing{} not working
with PigStorageSchema or PigStorage('\t', '-schema');
Posted by "David Ciemiewicz (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192279#comment-13192279 ]
David Ciemiewicz commented on PIG-2489:
---------------------------------------
Yes! PLEASE FIX THIS! Get it into Pig 0.10 ASAP.
> Input Path Globbing{} not working with PigStorageSchema or PigStorage('\t', '-schema');
> ---------------------------------------------------------------------------------------
>
> Key: PIG-2489
> URL: https://issues.apache.org/jira/browse/PIG-2489
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.1, 0.10
> Reporter: Mitesh Singh Jat
>
> {code:title=test.pig}
> -- For Pig 0.9
> --A = LOAD 'input/PigStorageSchema/Temp{1,2}/pss*' USING org.apache.pig.piggybank.storage.PigStorageSchema();
> -- For Pig 0.10
> A = LOAD 'input/PigStorageSchema/Temp{1,2}/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A
> {code}
> Schema file _input/PigStorageSchema/Temp{1,2}.pig_schema_
> {code}
> {"fields":[{"name":"name","type":55,"schema":null,"description":"autogenerated from Pig Field Schema"},{"name":"val","type":10,"schema":null,"description":"autogenerated from Pig Field Schema"}],"version":0,"sortKeys":[],"sortKeyOrders":[]}
> {code}
> Header file _input/PigStorageSchema/Temp{1,2}/.pig_header_
> {code}
> name val
> {code}
> Sample input file _input/PigStorageSchema/Temp1/pss.in_
> {code}
> peter 1
> samir 3
> michael 4
> peter 2
> peter 4
> samir 1
> {code}
> On running the above pig script _test.pig_ with pig 0.10, the following error is received.
> {noformat}
> 012-01-24 04:07:42,210 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> {noformat}
> Pig Stack Trace
> {noformat}
> Pig Stack Trace
> ---------------
> ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias A
> at org.apache.pig.PigServer.openIterator(PigServer.java:858)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:655)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:567)
> at org.apache.pig.Main.main(Main.java:111)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias A
> at org.apache.pig.PigServer.storeEx(PigServer.java:957)
> at org.apache.pig.PigServer.store(PigServer.java:920)
> at org.apache.pig.PigServer.openIterator(PigServer.java:833)
> ... 7 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2245:
> <file test.pig, line 2, column 4> Cannot get schema from loadFunc org.apache.pig.builtin.PigStorage
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:154)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:109)
> at org.apache.pig.newplan.logical.relational.LOStore.getSchema(LOStore.java:68)
> at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.validate(SchemaAliasVisitor.java:60)
> at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.visit(SchemaAliasVisitor.java:84)
> at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1618)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1612)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1335)
> at org.apache.pig.PigServer.storeEx(PigServer.java:952)
> ... 9 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> at org.apache.pig.builtin.JsonMetadata.nullOrException(JsonMetadata.java:222)
> at org.apache.pig.builtin.JsonMetadata.getSchema(JsonMetadata.java:191)
> at org.apache.pig.builtin.PigStorage.getSchema(PigStorage.java:438)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:150)
> {noformat}
> Whereas PigStorageSchema() or PigStorage('\t', '-schema') works with *wildcard* *.
> For example, following script works
> {code:title=test2.pig}
> A = LOAD 'input/PigStorageSchema/Temp*/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A;
> {code}
> As a workaround to make Temp{1,2} globbing work, the *,(comma)* separated multiple input paths (with no globbing)
> can given as input.
> {code:title=test2.pig}
> A = LOAD 'input/PigStorageSchema/Temp1/pss*,input/PigStorageSchema/Temp2/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A;
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira