You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org> on 2012/01/31 04:50:10 UTC
[jira] [Updated] (PIG-2489) Input Path Globbing{} not working with
PigStorageSchema or PigStorage('\t', '-schema');
[ https://issues.apache.org/jira/browse/PIG-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2489:
----------------------------
Attachment: PIG-2489-1.patch
> Input Path Globbing{} not working with PigStorageSchema or PigStorage('\t', '-schema');
> ---------------------------------------------------------------------------------------
>
> Key: PIG-2489
> URL: https://issues.apache.org/jira/browse/PIG-2489
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.1, 0.10
> Reporter: Mitesh Singh Jat
> Attachments: PIG-2489-1.patch
>
>
> {code:title=test.pig}
> -- For Pig 0.9
> --A = LOAD 'input/PigStorageSchema/Temp{1,2}/pss*' USING org.apache.pig.piggybank.storage.PigStorageSchema();
> -- For Pig 0.10
> A = LOAD 'input/PigStorageSchema/Temp{1,2}/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A
> {code}
> Schema file _input/PigStorageSchema/Temp{1,2}.pig_schema_
> {code}
> {"fields":[{"name":"name","type":55,"schema":null,"description":"autogenerated from Pig Field Schema"},{"name":"val","type":10,"schema":null,"description":"autogenerated from Pig Field Schema"}],"version":0,"sortKeys":[],"sortKeyOrders":[]}
> {code}
> Header file _input/PigStorageSchema/Temp{1,2}/.pig_header_
> {code}
> name val
> {code}
> Sample input file _input/PigStorageSchema/Temp1/pss.in_
> {code}
> peter 1
> samir 3
> michael 4
> peter 2
> peter 4
> samir 1
> {code}
> On running the above pig script _test.pig_ with pig 0.10, the following error is received.
> {noformat}
> 012-01-24 04:07:42,210 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> {noformat}
> Pig Stack Trace
> {noformat}
> Pig Stack Trace
> ---------------
> ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias A
> at org.apache.pig.PigServer.openIterator(PigServer.java:858)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:655)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:567)
> at org.apache.pig.Main.main(Main.java:111)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias A
> at org.apache.pig.PigServer.storeEx(PigServer.java:957)
> at org.apache.pig.PigServer.store(PigServer.java:920)
> at org.apache.pig.PigServer.openIterator(PigServer.java:833)
> ... 7 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2245:
> <file test.pig, line 2, column 4> Cannot get schema from loadFunc org.apache.pig.builtin.PigStorage
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:154)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:109)
> at org.apache.pig.newplan.logical.relational.LOStore.getSchema(LOStore.java:68)
> at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.validate(SchemaAliasVisitor.java:60)
> at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.visit(SchemaAliasVisitor.java:84)
> at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
> at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1618)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1612)
> at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1335)
> at org.apache.pig.PigServer.storeEx(PigServer.java:952)
> ... 9 more
> Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1131: Could not find schema file for hdfs://nameNode:8020/user/mitesh/input/PigStorageSchema/Temp{1,2}/pss*
> at org.apache.pig.builtin.JsonMetadata.nullOrException(JsonMetadata.java:222)
> at org.apache.pig.builtin.JsonMetadata.getSchema(JsonMetadata.java:191)
> at org.apache.pig.builtin.PigStorage.getSchema(PigStorage.java:438)
> at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:150)
> {noformat}
> Whereas PigStorageSchema() or PigStorage('\t', '-schema') works with *wildcard* *.
> For example, following script works
> {code:title=test2.pig}
> A = LOAD 'input/PigStorageSchema/Temp*/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A;
> {code}
> As a workaround to make Temp{1,2} globbing work, the *,(comma)* separated multiple input paths (with no globbing)
> can given as input.
> {code:title=test2.pig}
> A = LOAD 'input/PigStorageSchema/Temp1/pss*,input/PigStorageSchema/Temp2/pss*' USING PigStorage('\t', '-schema');
> DESCRIBE A;
> DUMP A;
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira