You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "mcouper99 (JIRA)" <ji...@apache.org> on 2013/12/04 15:28:35 UTC
[jira] [Commented] (PIG-3406) JsonStorage generates NPE after
successfully saving records
[ https://issues.apache.org/jira/browse/PIG-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13838924#comment-13838924 ]
mcouper99 commented on PIG-3406:
--------------------------------
I'm not sure if this is related, but I ran into the issue this morning. In looking at the generated schema, there does appear to be a 'null' name in the output:
{"fields":[{"name":null,"type":15,"description":"autogenerated from Pig Field Schema","schema":null},{"name":"group::valid_events::flight_id","type":15,"description":"autogenerated from Pig Field Schema","schema":null},{"name":"imps","type":15,"description":"autogenerated from Pig Field Schema","schema":null},{"name":"clicks","type":15,"description":"autogenerated from Pig Field Schema","schema":null}],"version":0,"sortKeys":[],"sortKeyOrders":[]}
In this case, I suspect it has something to do with the fact that I'm using a derived field in a group-by that doesn't have an associated alias:
group events by ((long)epoch/3600000 * 3600, flight_id);
> JsonStorage generates NPE after successfully saving records
> -----------------------------------------------------------
>
> Key: PIG-3406
> URL: https://issues.apache.org/jira/browse/PIG-3406
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.11.1
> Environment: OSX 10.8.4, running PigServer in local mode from a Groovy script.
> Reporter: Kirk Stork
>
> Make a new PigServer("local") instance, process some queries and save an alias using
> PigServer piggy = new PigServer("local")
> ... some queries..
> def job = piggy.store("c_observer_id", "obs.json", 'org.apache.pig.builtin.JsonStorage');
> The local directory obs.json is created and populated with correct results.
> but then an NPE is thrown, apparently during some attempt to store the schema.
> HadoopVersion PigVersion UserId StartedAt FinishedAt Features
> 1.1.0 0.11.1 kirk 2013-08-01 14:09:41 2013-08-01 14:09:42 GROUP_BY
> Success!
> Job Stats (time in seconds):
> JobId Alias Feature Outputs
> job_local_0001 c_observer_id,g_observer_id,obs GROUP_BY,COMBINER obs.json,
> Input(s):
> Successfully read records from: "/Volumes/Work/work/combatxxi-acceptance-testing/pig-input/Ambush_Mine_RPG-7.cxxi/Replication_1_SIMKIT_CONGRUENTIAL/ObserveLogger.log"
> Output(s):
> Successfully stored records in: "obs.json"
> Job DAG:
> job_local_0001
> org.apache.pig.PigException: ERROR 1002: Unable to store alias c_observer_id
> at org.apache.pig.PigServer.storeEx(PigServer.java:935)
> at org.apache.pig.PigServer.store(PigServer.java:898)
> at org.apache.pig.PigServer$store.call(Unknown Source)
> at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
> at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
> at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
> at edu.nps.cxxi.testbench.lff33.ObserveLoggerService.oink(ObserveLoggerService.groovy:27)
> at edu.nps.cxxi.testbench.lff33.ObserveLoggerService$oink.call(Unknown Source)
> at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
> at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
> at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
> at edu.nps.cxxi.testbench.lff33.ObserveLoggerServiceTests.testSomething(ObserveLoggerServiceTests.groovy:42)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
> at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:77)
> at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:195)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected error during execution.
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1275)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
> at org.apache.pig.PigServer.storeEx(PigServer.java:931)
> ... 41 more
> Caused by: java.lang.NullPointerException
> at org.apache.pig.builtin.JsonMetadata.storeSchema(JsonMetadata.java:314)
> at org.apache.pig.builtin.JsonStorage.storeSchema(JsonStorage.java:274)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.storeCleanup(PigOutputCommitter.java:141)
> at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.storeSchemaForLocal(HadoopShims.java:85)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:427)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
> ... 43 more
> 13/08/01 14:09:42 INFO spring.GrailsWebApplicationContext: Closing org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext@784ed19b: startup date [Thu Aug 01 14:09:39 PDT 2013]; root of context hierarchy
> 13/08/01 14:09:42 INFO support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4d838b: defining beans [grailsApplication,pluginManager,messageSource,orgGrailsBeansConstraintsEvaluator,grailsProxyHandler,observeLoggerService]; root of factory hierarchy
> If I save to PigStorage, it works fine.
--
This message was sent by Atlassian JIRA
(v6.1#6144)