You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Meghana Narasimhan <Me...@demandmedia.com> on 2012/07/31 21:48:23 UTC
Pig AvroStorage
Hi ,
I am using pig-0.9.2 with avro storage. When I try to use an external schema file to store my output, I get an exception with :
Caused by: java.io.IOException: Invalid parameter:schema_file
I followed the code at:
http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
Here is my schema file :
1.
{
2.
"type":"record",
3.
"name":"TUPLE_2",
4.
"fields":[
5.
{"name":"logTimestampMs","type":["null","long"]},
6.
{"name":"queueTimestampMs","type":["null","long"]}
7.
]
8.
}
9.
10.
11.
Here is my Pig Storage statement :
12.
13.
store imp into 'output' using org.apache.pig.piggybank.storage.avro.AvroStorage('{"schema_file":"/Users/xxxx/test.avsc","field0":"def:logTimestampMs","field1":"def:queueTimestampMs"}');
And here is my exception:
ERROR 1200: could not instantiate 'org.apache.pig.piggybank.storage.avro.AvroStorage' with arguments '[{"schema_file":"/user/xxxx/test.avsc","field0":"def:logTimestampMs","field1":"def:queueTimestampMs"}]'
Failed to parse: could not instantiate 'org.apache.pig.piggybank.storage.avro.AvroStorage' with arguments '[{"schema_file":"/user/xxxx/test.avsc","field0":"def:logTimestampMs","field1":"def:queueTimestampMs"}]'
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:180)
at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:495)
at org.apache.pig.Main.main(Main.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.RuntimeException: could not instantiate 'org.apache.pig.piggybank.storage.avro.AvroStorage' with arguments '[{"schema_file":"/user/xxxx/test.avsc","field0":"def:logTimestampMs","field1":"def:queueTimestampMs"}]'
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:493)
at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:723)
at org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.java:712)
at org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.java:4340)
at org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator.java:5956)
at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1122)
at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:683)
at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:483)
at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:369)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:171)
... 15 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:483)
... 24 more
Caused by: java.io.IOException: Invalid parameter:schema_file
at org.apache.pig.piggybank.storage.avro.AvroStorage.init(AvroStorage.java:447)
at org.apache.pig.piggybank.storage.avro.AvroStorage.<init>(AvroStorage.java:120)
... 28 more
Thanks,
Meg
________________________________
Please NOTE: This electronic message, including any attachments, may include privileged, confidential and/or inside information owned by Demand Media, Inc. Any distribution or use of this communication by anyone other than the intended recipient(s) is strictly prohibited and may be unlawful. If you are not the intended recipient, please notify the sender by replying to this message and then delete it from your system. Thank you.
Re: Pig AvroStorage
Posted by Russell Jurney <ru...@gmail.com>.
Try Pig 0.10, and it will probably work.
Russell Jurney http://datasyndrome.com
On Jul 31, 2012, at 1:32 PM, Meghana Narasimhan
<Me...@demandmedia.com> wrote:
> Hi ,
>
> I am using pig-0.9.2 with avro storage. When I try to use an external schema file to store my output, I get an exception with :
>
> Caused by: java.io.IOException: Invalid parameter:schema_file
>
> I followed the code at:
>
> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
>
> Here is my schema file :
>
>
> 1.
> {
> 2.
> "type":"record",
> 3.
> "name":"TUPLE_2",
> 4.
> "fields":[
> 5.
> {"name":"logTimestampMs","type":["null","long"]},
> 6.
> {"name":"queueTimestampMs","type":["null","long"]}
> 7.
> ]
> 8.
> }
> 9.
>
> 10.
>
> 11.
> Here is my Pig Storage statement :
> 12.
>
> 13.
> store imp into 'output' using org.apache.pig.piggybank.storage.avro.AvroStorage('{"schema_file":"/Users/xxxx/test.avsc","field0":"def:logTimestampMs","field1":"def:queueTimestampMs"}');
>
> And here is my exception:
>
> ERROR 1200: could not instantiate 'org.apache.pig.piggybank.storage.avro.AvroStorage' with arguments '[{"schema_file":"/user/xxxx/test.avsc","field0":"def:logTimestampMs","field1":"def:queueTimestampMs"}]'
>
> Failed to parse: could not instantiate 'org.apache.pig.piggybank.storage.avro.AvroStorage' with arguments '[{"schema_file":"/user/xxxx/test.avsc","field0":"def:logTimestampMs","field1":"def:queueTimestampMs"}]'
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:180)
> at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> at org.apache.pig.Main.run(Main.java:495)
> at org.apache.pig.Main.main(Main.java:111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.lang.RuntimeException: could not instantiate 'org.apache.pig.piggybank.storage.avro.AvroStorage' with arguments '[{"schema_file":"/user/xxxx/test.avsc","field0":"def:logTimestampMs","field1":"def:queueTimestampMs"}]'
> at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:493)
> at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:723)
> at org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.java:712)
> at org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.java:4340)
> at org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator.java:5956)
> at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1122)
> at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:683)
> at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:483)
> at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:369)
> at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:171)
> ... 15 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:483)
> ... 24 more
> Caused by: java.io.IOException: Invalid parameter:schema_file
> at org.apache.pig.piggybank.storage.avro.AvroStorage.init(AvroStorage.java:447)
> at org.apache.pig.piggybank.storage.avro.AvroStorage.<init>(AvroStorage.java:120)
> ... 28 more
>
>
> Thanks,
> Meg
>
> ________________________________
> Please NOTE: This electronic message, including any attachments, may include privileged, confidential and/or inside information owned by Demand Media, Inc. Any distribution or use of this communication by anyone other than the intended recipient(s) is strictly prohibited and may be unlawful. If you are not the intended recipient, please notify the sender by replying to this message and then delete it from your system. Thank you.