Re: Exception during parallel jobs execution using python

[Jakub Glapa <ja...@gmail.com>]

Hi Cheolsoo,
it's been a while but I finally filled a jira bug.
Unfortunately I'm not able to provide a reproducible use case.
I've described the problem I think in much detail.
Maybe somebody more experienced would know where to look for the root cause
of the problem or would recommend some debug techniques.

Thanks!

https://issues.apache.org/jira/browse/PIG-3263


--
regards,
Jakub Glapa


On Fri, Feb 15, 2013 at 10:05 PM, Cheolsoo Park <ch...@cloudera.com>wrote:

> Hi Jakub,
>
> >> About 5 jobs fail like that out of 150.
>
> It sounds like a multi-threading bug in Pig to me. Do you mind filing a
> jira and attaching your script that reproduces the issue?
>
> Thanks,
> Cheolsoo
>
>
> On Fri, Feb 15, 2013 at 8:27 AM, Jakub Glapa <ja...@gmail.com>
> wrote:
>
> > I have a set of jobs to run with different parameters.
> >  I'm using Python to prepare the parameter sets and then I'm executing
> them
> > in batches with Pig.run(batchOfParams).
> >
> > The number of jobs is quite high but I'm running 4 of them at a time (4
> > sets of params in batchOfParams).
> >
> > From time to time I started noticing exception that terminate some of the
> > jobs.
> >
> > Can it be a problem with my UDF or is there a bug in in the Pig.run
> logic?
> >
> > The OrderQueryTokens is trivial. It just orders the words in string.
> >
> > About 5 jobs fail like that out of 150.
> >
> >
> > exception:
> >
> >
> > 2013-02-14 16:25:04,575 [main] ERROR
> org.apache.pig.scripting.BoundScript -
> > Pig pipeline failed to complete
> > java.util.concurrent.ExecutionException:
> > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error
> > during parsing. Could not resolve my.pig.udf.OrderQueryTokens using
> > imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
> > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> > at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> > at org.apache.pig.scripting.BoundScript.run(BoundScript.java:176)
> > at org.apache.pig.scripting.BoundScript.run(BoundScript.java:134)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:186)
> > at
> >
> org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:204)
> > at org.python.core.PyObject.__call__(PyObject.java:387)
> > at org.python.core.PyObject.__call__(PyObject.java:391)
> > at org.python.core.PyMethod.__call__(PyMethod.java:109)
> > at
> >
> >
> org.python.pycode._pyx3.main$1(/data/servers/autocomplete_pig/scripts/run.py:65)
> > at
> >
> >
> org.python.pycode._pyx3.call_function(/data/servers/autocomplete_pig/scripts/run.py)
> > at org.python.core.PyTableCode.call(PyTableCode.java:165)
> > at org.python.core.PyBaseCode.call(PyBaseCode.java:120)
> > at org.python.core.PyFunction.__call__(PyFunction.java:307)
> > at
> >
> >
> org.python.pycode._pyx4.f$0(/data/servers/autocomplete_pig/scripts/run.py:210)
> > at
> >
> >
> org.python.pycode._pyx4.call_function(/data/servers/autocomplete_pig/scripts/run.py)
> > at org.python.core.PyTableCode.call(PyTableCode.java:165)
> > at org.python.core.PyCode.call(PyCode.java:18)
> > at org.python.core.Py.runCode(Py.java:1275)
> > at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:235)
> > at
> >
> >
> org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:199)
> > at
> >
> >
> org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:396)
> > at
> >
> >
> org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:380)
> > at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:295)
> > at org.apache.pig.Main.runEmbeddedScript(Main.java:929)
> > at org.apache.pig.Main.run(Main.java:510)
> > at org.apache.pig.Main.main(Main.java:111)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR
> 1000:
> > Error during parsing. Could not resolve my.pig.udf.OrderQueryTokens using
> > imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
> > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1606)
> > at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1549)
> > at org.apache.pig.PigServer.registerQuery(PigServer.java:549)
> > at
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:971)
> > at
> >
> >
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
> > at
> >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:190)
> > at
> >
> org.apache.pig.scripting.BoundScript$MyCallable.call(BoundScript.java:342)
> > at
> >
> org.apache.pig.scripting.BoundScript$MyCallable.call(BoundScript.java:318)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > Caused by: Failed to parse: Pig script failed to parse:
> > <line 36, column 76> Failed to generate logical plan. Nested exception:
> > java.lang.RuntimeException: Cannot instantiate:
> my.pig.udf.OrderQueryTokens
> > at
> > org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
> > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1598)
> > ... 12 more
> > Caused by:
> > <line 36, column 76> Failed to generate logical plan. Nested exception:
> > java.lang.RuntimeException: Cannot instantiate:
> my.pig.udf.OrderQueryTokens
> > at
> >
> >
> org.apache.pig.parser.LogicalPlanBuilder.buildUDF(LogicalPlanBuilder.java:980)
> > at
> >
> >
> org.apache.pig.parser.LogicalPlanGenerator.func_eval(LogicalPlanGenerator.java:7316)
> > at
> >
> >
> org.apache.pig.parser.LogicalPlanGenerator.projectable_expr(LogicalPlanGenerator.java:8857)
> > at
> >
> >
> org.apache.pig.parser.LogicalPlanGenerator.var_expr(LogicalPlanGenerator.java:8632)
> > at
> >
> >
> org.apache.pig.parser.LogicalPlanGenerator.expr(LogicalPlanGenerator.java:7984)
> > at
> >
> >
> org.apache.pig.parser.LogicalPlanGenerator.flatten_generated_item(LogicalPlanGenerator.java:5962)
> > at
> >
> >
> org.apache.pig.parser.LogicalPlanGenerator.generate_clause(LogicalPlanGenerator.java:14101)
> > at
> >
> >
> org.apache.pig.parser.LogicalPlanGenerator.foreach_plan(LogicalPlanGenerator.java:12493)
> > at
> >
> >
> org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:12360)
> > at
> >
> >
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1577)
> > at
> >
> >
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:789)
> > at
> >
> >
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:507)
> > at
> >
> >
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:382)
> > at
> > org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:177)
> > ... 13 more
> > Caused by: java.lang.RuntimeException: Cannot instantiate:
> > my.pig.udf.OrderQueryTokens
> > at
> >
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:511)
> > at
> >
> >
> org.apache.pig.impl.PigContext.instantiateFuncFromAlias(PigContext.java:581)
> > at
> >
> >
> org.apache.pig.parser.LogicalPlanBuilder.buildUDF(LogicalPlanBuilder.java:977)
> > ... 26 more
> > Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
> > 1070: Could not resolve my.pig.udf.OrderQueryTokens using imports: [,
> > org.apache.pig.builtin., org.apache.pig.impl.builtin.]
> > at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:496)
> > at
> >
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:508)
> > ... 28 more
> >
> >
> >
> >
> >
> > --
> > regards,
> > Jakub Glapa
> >
>