You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Dan Zhu <da...@yahoo-inc.com> on 2013/07/24 23:12:42 UTC

Parse Nested JSON String in Pig

Hi pig-users,

I have tuples of nested JSON string, I am trying to parse it in Pig.

I tried com.twitter.elephantbird.pig.piggybank.JsonStringToMap() (acctDataJsonized = FOREACH acctData GENERATE com.mozilla.pig.eval.json.JsonMap($0) AS json_map:map[];) but it only parsed the first level rather than nested.

I also tried com.mozilla.pig.eval.json.JsonMap() (acctDataJsonized = FOREACH acctData GENERATE com.twitter.elephantbird.pig.piggybank.JsonStringToMap($0);) but it throws: (is it due to NULL value in the JSON data?)

ERROR 1200: Pig script failed to parse:
<line 7, column 45> Failed to generate logical plan. Nested exception: java.lang.RuntimeException: could not instantiate 'com.mozilla.pig.eval.json.JsonMap' with arguments 'null'

Failed to parse: Pig script failed to parse:
<line 7, column 45> Failed to generate logical plan. Nested exception: java.lang.RuntimeException: could not instantiate 'com.mozilla.pig.eval.json.JsonMap' with arguments 'null'
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1570)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1543)
at org.apache.pig.PigServer.registerQuery(PigServer.java:545)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:970)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:495)
at org.apache.pig.Main.main(Main.java:111)
Caused by:
<line 7, column 45> Failed to generate logical plan. Nested exception: java.lang.RuntimeException: could not instantiate 'com.mozilla.pig.eval.json.JsonMap' with arguments 'null'
at org.apache.pig.parser.LogicalPlanBuilder.buildUDF(LogicalPlanBuilder.java:980)
at org.apache.pig.parser.LogicalPlanGenerator.func_eval(LogicalPlanGenerator.java:7316)
at org.apache.pig.parser.LogicalPlanGenerator.projectable_expr(LogicalPlanGenerator.java:8857)
at org.apache.pig.parser.LogicalPlanGenerator.var_expr(LogicalPlanGenerator.java:8632)
at org.apache.pig.parser.LogicalPlanGenerator.expr(LogicalPlanGenerator.java:7984)
at org.apache.pig.parser.LogicalPlanGenerator.flatten_generated_item(LogicalPlanGenerator.java:5962)
at org.apache.pig.parser.LogicalPlanGenerator.generate_clause(LogicalPlanGenerator.java:14101)
at org.apache.pig.parser.LogicalPlanGenerator.foreach_plan(LogicalPlanGenerator.java:12493)
at org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:12360)
at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1577)
at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:789)
at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:507)
at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:382)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:177)
... 10 more
Caused by: java.lang.RuntimeException: could not instantiate 'com.mozilla.pig.eval.json.JsonMap' with arguments 'null'
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:554)
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:560)
at org.apache.pig.impl.PigContext.instantiateFuncFromAlias(PigContext.java:583)
at org.apache.pig.parser.LogicalPlanBuilder.buildUDF(LogicalPlanBuilder.java:977)
... 23 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.fasterxml.jackson.databind.ObjectMapper
at com.mozilla.pig.eval.json.JsonMap.<init>(JsonMap.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at java.lang.Class.newInstance0(Class.java:374)
at java.lang.Class.newInstance(Class.java:327)
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:524)
... 26 more

Any suggestion will be very appreciated.

Thanks & Regards,

Dan

Re: Parse Nested JSON String in Pig

Posted by Harsha <ha...@defun.org>.
Dan,  
    include these jars  
register 'jackson-core-2.0.6.jar'
register 'jackson-databind-2.0.6.jar'
register 'jackson-annotations-2.0.6.jar'




--  
Harsha


On Wednesday, July 24, 2013 at 2:25 PM, Serega Sheypak wrote:

> There is a missing dependency, a jar with class
> com.fasterxml.jackson.databind.ObjectMapper
> Use Google to find jar. Suggest you to use maven public repos.
> 24.07.2013 23:16 пользователь "Dan Zhu" <danzhu@yahoo-inc.com (mailto:danzhu@yahoo-inc.com)> написал:
>  
> > Hi pig-users,
> >  
> > I have tuples of nested JSON string, I am trying to parse it in Pig.
> >  
> > I tried com.twitter.elephantbird.pig.piggybank.JsonStringToMap()
> > (acctDataJsonized = FOREACH acctData GENERATE
> > com.mozilla.pig.eval.json.JsonMap($0) AS json_map:map[];) but it only
> > parsed the first level rather than nested.
> >  
> > I also tried com.mozilla.pig.eval.json.JsonMap() (acctDataJsonized =
> > FOREACH acctData GENERATE
> > com.twitter.elephantbird.pig.piggybank.JsonStringToMap($0);) but it throws:
> > (is it due to NULL value in the JSON data?)
> >  
> > ERROR 1200: Pig script failed to parse:
> > <line 7, column 45> Failed to generate logical plan. Nested exception:
> > java.lang.RuntimeException: could not instantiate
> > 'com.mozilla.pig.eval.json.JsonMap' with arguments 'null'
> >  
> > Failed to parse: Pig script failed to parse:
> > <line 7, column 45> Failed to generate logical plan. Nested exception:
> > java.lang.RuntimeException: could not instantiate
> > 'com.mozilla.pig.eval.json.JsonMap' with arguments 'null'
> > at
> > org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
> > at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1570)
> > at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1543)
> > at org.apache.pig.PigServer.registerQuery(PigServer.java:545)
> > at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:970)
> > at
> > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
> > at
> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
> > at
> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> > at org.apache.pig.Main.run(Main.java:495)
> > at org.apache.pig.Main.main(Main.java:111)
> > Caused by:
> > <line 7, column 45> Failed to generate logical plan. Nested exception:
> > java.lang.RuntimeException: could not instantiate
> > 'com.mozilla.pig.eval.json.JsonMap' with arguments 'null'
> > at
> > org.apache.pig.parser.LogicalPlanBuilder.buildUDF(LogicalPlanBuilder.java:980)
> > at
> > org.apache.pig.parser.LogicalPlanGenerator.func_eval(LogicalPlanGenerator.java:7316)
> > at
> > org.apache.pig.parser.LogicalPlanGenerator.projectable_expr(LogicalPlanGenerator.java:8857)
> > at
> > org.apache.pig.parser.LogicalPlanGenerator.var_expr(LogicalPlanGenerator.java:8632)
> > at
> > org.apache.pig.parser.LogicalPlanGenerator.expr(LogicalPlanGenerator.java:7984)
> > at
> > org.apache.pig.parser.LogicalPlanGenerator.flatten_generated_item(LogicalPlanGenerator.java:5962)
> > at
> > org.apache.pig.parser.LogicalPlanGenerator.generate_clause(LogicalPlanGenerator.java:14101)
> > at
> > org.apache.pig.parser.LogicalPlanGenerator.foreach_plan(LogicalPlanGenerator.java:12493)
> > at
> > org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:12360)
> > at
> > org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1577)
> > at
> > org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:789)
> > at
> > org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:507)
> > at
> > org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:382)
> > at
> > org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:177)
> > ... 10 more
> > Caused by: java.lang.RuntimeException: could not instantiate
> > 'com.mozilla.pig.eval.json.JsonMap' with arguments 'null'
> > at
> > org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:554)
> > at
> > org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:560)
> > at
> > org.apache.pig.impl.PigContext.instantiateFuncFromAlias(PigContext.java:583)
> > at
> > org.apache.pig.parser.LogicalPlanBuilder.buildUDF(LogicalPlanBuilder.java:977)
> > ... 23 more
> > Caused by: java.lang.NoClassDefFoundError: Could not initialize class
> > com.fasterxml.jackson.databind.ObjectMapper
> > at com.mozilla.pig.eval.json.JsonMap.<init>(JsonMap.java:47)
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> > at
> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> > at
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> > at java.lang.Class.newInstance0(Class.java:374)
> > at java.lang.Class.newInstance(Class.java:327)
> > at
> > org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:524)
> > ... 26 more
> >  
> > Any suggestion will be very appreciated.
> >  
> > Thanks & Regards,
> >  
> > Dan

Re: Parse Nested JSON String in Pig

Posted by Serega Sheypak <se...@gmail.com>.
There is a missing dependency, a jar with class
com.fasterxml.jackson.databind.ObjectMapper
Use Google to find jar. Suggest you to use maven public repos.
24.07.2013 23:16 пользователь "Dan Zhu" <da...@yahoo-inc.com> написал:

> Hi pig-users,
>
> I have tuples of nested JSON string, I am trying to parse it in Pig.
>
> I tried com.twitter.elephantbird.pig.piggybank.JsonStringToMap()
> (acctDataJsonized = FOREACH acctData GENERATE
> com.mozilla.pig.eval.json.JsonMap($0) AS json_map:map[];) but it only
> parsed the first level rather than nested.
>
> I also tried com.mozilla.pig.eval.json.JsonMap() (acctDataJsonized =
> FOREACH acctData GENERATE
> com.twitter.elephantbird.pig.piggybank.JsonStringToMap($0);) but it throws:
> (is it due to NULL value in the JSON data?)
>
> ERROR 1200: Pig script failed to parse:
> <line 7, column 45> Failed to generate logical plan. Nested exception:
> java.lang.RuntimeException: could not instantiate
> 'com.mozilla.pig.eval.json.JsonMap' with arguments 'null'
>
> Failed to parse: Pig script failed to parse:
> <line 7, column 45> Failed to generate logical plan. Nested exception:
> java.lang.RuntimeException: could not instantiate
> 'com.mozilla.pig.eval.json.JsonMap' with arguments 'null'
> at
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
> at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1570)
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1543)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:545)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:970)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> at org.apache.pig.Main.run(Main.java:495)
> at org.apache.pig.Main.main(Main.java:111)
> Caused by:
> <line 7, column 45> Failed to generate logical plan. Nested exception:
> java.lang.RuntimeException: could not instantiate
> 'com.mozilla.pig.eval.json.JsonMap' with arguments 'null'
> at
> org.apache.pig.parser.LogicalPlanBuilder.buildUDF(LogicalPlanBuilder.java:980)
> at
> org.apache.pig.parser.LogicalPlanGenerator.func_eval(LogicalPlanGenerator.java:7316)
> at
> org.apache.pig.parser.LogicalPlanGenerator.projectable_expr(LogicalPlanGenerator.java:8857)
> at
> org.apache.pig.parser.LogicalPlanGenerator.var_expr(LogicalPlanGenerator.java:8632)
> at
> org.apache.pig.parser.LogicalPlanGenerator.expr(LogicalPlanGenerator.java:7984)
> at
> org.apache.pig.parser.LogicalPlanGenerator.flatten_generated_item(LogicalPlanGenerator.java:5962)
> at
> org.apache.pig.parser.LogicalPlanGenerator.generate_clause(LogicalPlanGenerator.java:14101)
> at
> org.apache.pig.parser.LogicalPlanGenerator.foreach_plan(LogicalPlanGenerator.java:12493)
> at
> org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:12360)
> at
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1577)
> at
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:789)
> at
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:507)
> at
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:382)
> at
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:177)
> ... 10 more
> Caused by: java.lang.RuntimeException: could not instantiate
> 'com.mozilla.pig.eval.json.JsonMap' with arguments 'null'
> at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:554)
> at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:560)
> at
> org.apache.pig.impl.PigContext.instantiateFuncFromAlias(PigContext.java:583)
> at
> org.apache.pig.parser.LogicalPlanBuilder.buildUDF(LogicalPlanBuilder.java:977)
> ... 23 more
> Caused by: java.lang.NoClassDefFoundError: Could not initialize class
> com.fasterxml.jackson.databind.ObjectMapper
> at com.mozilla.pig.eval.json.JsonMap.<init>(JsonMap.java:47)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at java.lang.Class.newInstance0(Class.java:374)
> at java.lang.Class.newInstance(Class.java:327)
> at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:524)
> ... 26 more
>
> Any suggestion will be very appreciated.
>
> Thanks & Regards,
>
> Dan
>