You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Vincent Barat <vi...@ubikod.com> on 2010/02/25 16:04:00 UTC

PIG 0.6.0 branch: reentrance errors even with multiple PigServer

Hi,

I'm facing what seem to be re-entrance errors when using PIG through 
the Java API.

I know that the PigServer object is not reentrant, so I instantiate 
several PigServers and run them in separated threads. Even in that 
case, I face the infamous exception "Unable to find clone for op 
Project 5-482 Projections: [3] Overloaded: false".

Is there other reentrance issues in the PIG library?

2010-02-25 15:49:06,128 | ERROR | Errors | stderr     | Caused by: 
java.lang.RuntimeException: Unable to find clone for op Project 
5-482 Projections: [3] Overloaded: false
2010-02-25 15:49:06,128 | ERROR | Errors | stderr     | 	at 
org.apache.pig.impl.logicalLayer.LogicalPlan.clone(LogicalPlan.java:139)
2010-02-25 15:49:06,129 | ERROR | Errors | stderr     | 	at 
org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.<init>(LogicalPlanCloneHelper.java:63)
2010-02-25 15:49:06,129 | ERROR | Errors | stderr     | 	at 
org.apache.pig.impl.logicalLayer.LogicalPlanCloner.getClonedPlan(LogicalPlanCloner.java:45)
2010-02-25 15:49:06,129 | ERROR | Errors | stderr     | 	at 
org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:3083)
2010-02-25 15:49:06,129 | ERROR | Errors | stderr     | 	at 
org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1370)
2010-02-25 15:49:06,130 | ERROR | Errors | stderr     | 	at 
org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:949)
2010-02-25 15:49:06,130 | ERROR | Errors | stderr     | 	at 
org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:738)
2010-02-25 15:49:06,130 | ERROR | Errors | stderr     | 	at 
org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
2010-02-25 15:49:06,130 | ERROR | Errors | stderr     | 	at 
org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1036)
2010-02-25 15:49:06,130 | ERROR | Errors | stderr     | 	at 
org.apache.pig.PigServer$Graph.clone(PigServer.java:1079)
2010-02-25 15:49:06,130 | ERROR | Errors | stderr     | 	at 
org.apache.pig.PigServer.clonePlan(PigServer.java:390)
2010-02-25 15:49:06,130 | ERROR | Errors | stderr     | 	at 
org.apache.pig.PigServer.compileLp(PigServer.java:806)
2010-02-25 15:49:06,131 | ERROR | Errors | stderr     | 	at 
org.apache.pig.PigServer.compileLp(PigServer.java:793)
2010-02-25 15:49:06,131 | ERROR | Errors | stderr     | 	at 
org.apache.pig.PigServer.store(PigServer.java:510)

Re: PIG 0.6.0 branch: reentrance errors even with multiple PigServer

Posted by Vincent Barat <vi...@ubikod.com>.
I've made a small modification in the last patch and submitted a 
correction (but I really don't understand how the code work, so...).

It works for me (no more exception) and I'm currently running the 
test suite to check it.

My guess is that there is yet some work to do to integrate all 
proposed patches in one global correction, but it would be great to 
do it ;-)

Le 25/02/10 16:55, Vincent Barat a écrit :
> Yes, I 've tried the last one, but it does not work for me:
>
> I investigate...
>
> 2010-02-25 16:46:03,679 | FATAL | Users | Main | Uncaught handler:
> thread Users (17) exiting due to uncaught exception
> java.lang.NullPointerException
> at
> org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.visit(LogicalPlanCloneHelper.java:331)
>
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at
> org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:69)
>
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at
> org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.getClonedPlan(LogicalPlanCloneHelper.java:75)
>
> at org.apache.pig.impl.logicalLayer.LOSort.clone(LOSort.java:213)
> at org.apache.pig.impl.logicalLayer.LogicalPlan.clone(LogicalPlan.java:115)
> at
> org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.<init>(LogicalPlanCloneHelper.java:63)
>
> at
> org.apache.pig.impl.logicalLayer.LogicalPlanCloner.getClonedPlan(LogicalPlanCloner.java:45)
>
> at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:3083)
>
> at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1370)
>
> at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:949)
>
> at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:738)
>
> at
> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
>
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1036)
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:986)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:386)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:402)
>
>
>
> Le 25/02/10 16:29, Jeff Zhang a écrit :
>> Hi Vincent,
>>
>> Pig do not support multi-thread officially. But you can try patches here
>> https://issues.apache.org/jira/browse/PIG-240
>>
>>
>> On Thu, Feb 25, 2010 at 7:04 AM, Vincent
>> Barat<vi...@ubikod.com>wrote:
>>
>>> Hi,
>>>
>>> I'm facing what seem to be re-entrance errors when using PIG through the
>>> Java API.
>>>
>>> I know that the PigServer object is not reentrant, so I instantiate
>>> several
>>> PigServers and run them in separated threads. Even in that case, I
>>> face the
>>> infamous exception "Unable to find clone for op Project 5-482
>>> Projections:
>>> [3] Overloaded: false".
>>>
>>> Is there other reentrance issues in the PIG library?
>>>
>>> 2010-02-25 15:49:06,128 | ERROR | Errors | stderr | Caused by:
>>> java.lang.RuntimeException: Unable to find clone for op Project 5-482
>>> Projections: [3] Overloaded: false
>>> 2010-02-25 15:49:06,128 | ERROR | Errors | stderr | at
>>> org.apache.pig.impl.logicalLayer.LogicalPlan.clone(LogicalPlan.java:139)
>>> 2010-02-25 15:49:06,129 | ERROR | Errors | stderr | at
>>> org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.<init>(LogicalPlanCloneHelper.java:63)
>>>
>>> 2010-02-25 15:49:06,129 | ERROR | Errors | stderr | at
>>> org.apache.pig.impl.logicalLayer.LogicalPlanCloner.getClonedPlan(LogicalPlanCloner.java:45)
>>>
>>> 2010-02-25 15:49:06,129 | ERROR | Errors | stderr | at
>>> org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:3083)
>>>
>>> 2010-02-25 15:49:06,129 | ERROR | Errors | stderr | at
>>> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1370)
>>>
>>> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr | at
>>> org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:949)
>>>
>>> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr | at
>>> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:738)
>>>
>>> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr | at
>>> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
>>>
>>> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr | at
>>> org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1036)
>>> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr | at
>>> org.apache.pig.PigServer$Graph.clone(PigServer.java:1079)
>>> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr | at
>>> org.apache.pig.PigServer.clonePlan(PigServer.java:390)
>>> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr | at
>>> org.apache.pig.PigServer.compileLp(PigServer.java:806)
>>> 2010-02-25 15:49:06,131 | ERROR | Errors | stderr | at
>>> org.apache.pig.PigServer.compileLp(PigServer.java:793)
>>> 2010-02-25 15:49:06,131 | ERROR | Errors | stderr | at
>>> org.apache.pig.PigServer.store(PigServer.java:510)
>>>
>>
>>
>>

Re: PIG 0.6.0 branch: reentrance errors even with multiple PigServer

Posted by Vincent Barat <vi...@ubikod.com>.
Yes, I 've tried the last one, but it does not work for me:

I investigate...

2010-02-25 16:46:03,679 | FATAL | Users | Main       | Uncaught 
handler: thread Users (17) exiting due to uncaught exception
java.lang.NullPointerException
	at 
org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.visit(LogicalPlanCloneHelper.java:331)
	at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
	at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
	at 
org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:69)
	at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
	at 
org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.getClonedPlan(LogicalPlanCloneHelper.java:75)
	at org.apache.pig.impl.logicalLayer.LOSort.clone(LOSort.java:213)
	at 
org.apache.pig.impl.logicalLayer.LogicalPlan.clone(LogicalPlan.java:115)
	at 
org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.<init>(LogicalPlanCloneHelper.java:63)
	at 
org.apache.pig.impl.logicalLayer.LogicalPlanCloner.getClonedPlan(LogicalPlanCloner.java:45)
	at 
org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:3083)
	at 
org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1370)
	at 
org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:949)
	at 
org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:738)
	at 
org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
	at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1036)
	at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:986)
	at org.apache.pig.PigServer.registerQuery(PigServer.java:386)
	at org.apache.pig.PigServer.registerQuery(PigServer.java:402)



Le 25/02/10 16:29, Jeff Zhang a écrit :
> Hi Vincent,
>
> Pig do not support multi-thread officially. But you can try patches here
> https://issues.apache.org/jira/browse/PIG-240
>
>
> On Thu, Feb 25, 2010 at 7:04 AM, Vincent Barat<vi...@ubikod.com>wrote:
>
>> Hi,
>>
>> I'm facing what seem to be re-entrance errors when using PIG through the
>> Java API.
>>
>> I know that the PigServer object is not reentrant, so I instantiate several
>> PigServers and run them in separated threads. Even in that case, I face the
>> infamous exception "Unable to find clone for op Project 5-482 Projections:
>> [3] Overloaded: false".
>>
>> Is there other reentrance issues in the PIG library?
>>
>> 2010-02-25 15:49:06,128 | ERROR | Errors | stderr     | Caused by:
>> java.lang.RuntimeException: Unable to find clone for op Project 5-482
>> Projections: [3] Overloaded: false
>> 2010-02-25 15:49:06,128 | ERROR | Errors | stderr     |         at
>> org.apache.pig.impl.logicalLayer.LogicalPlan.clone(LogicalPlan.java:139)
>> 2010-02-25 15:49:06,129 | ERROR | Errors | stderr     |         at
>> org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.<init>(LogicalPlanCloneHelper.java:63)
>> 2010-02-25 15:49:06,129 | ERROR | Errors | stderr     |         at
>> org.apache.pig.impl.logicalLayer.LogicalPlanCloner.getClonedPlan(LogicalPlanCloner.java:45)
>> 2010-02-25 15:49:06,129 | ERROR | Errors | stderr     |         at
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:3083)
>> 2010-02-25 15:49:06,129 | ERROR | Errors | stderr     |         at
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1370)
>> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr     |         at
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:949)
>> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr     |         at
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:738)
>> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr     |         at
>> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
>> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr     |         at
>> org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1036)
>> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr     |         at
>> org.apache.pig.PigServer$Graph.clone(PigServer.java:1079)
>> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr     |         at
>> org.apache.pig.PigServer.clonePlan(PigServer.java:390)
>> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr     |         at
>> org.apache.pig.PigServer.compileLp(PigServer.java:806)
>> 2010-02-25 15:49:06,131 | ERROR | Errors | stderr     |         at
>> org.apache.pig.PigServer.compileLp(PigServer.java:793)
>> 2010-02-25 15:49:06,131 | ERROR | Errors | stderr     |         at
>> org.apache.pig.PigServer.store(PigServer.java:510)
>>
>
>
>

Re: PIG 0.6.0 branch: reentrance errors even with multiple PigServer

Posted by Jeff Zhang <zj...@gmail.com>.
Hi Vincent,

Pig do not support multi-thread officially. But you can try patches here
https://issues.apache.org/jira/browse/PIG-240


On Thu, Feb 25, 2010 at 7:04 AM, Vincent Barat <vi...@ubikod.com>wrote:

> Hi,
>
> I'm facing what seem to be re-entrance errors when using PIG through the
> Java API.
>
> I know that the PigServer object is not reentrant, so I instantiate several
> PigServers and run them in separated threads. Even in that case, I face the
> infamous exception "Unable to find clone for op Project 5-482 Projections:
> [3] Overloaded: false".
>
> Is there other reentrance issues in the PIG library?
>
> 2010-02-25 15:49:06,128 | ERROR | Errors | stderr     | Caused by:
> java.lang.RuntimeException: Unable to find clone for op Project 5-482
> Projections: [3] Overloaded: false
> 2010-02-25 15:49:06,128 | ERROR | Errors | stderr     |         at
> org.apache.pig.impl.logicalLayer.LogicalPlan.clone(LogicalPlan.java:139)
> 2010-02-25 15:49:06,129 | ERROR | Errors | stderr     |         at
> org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.<init>(LogicalPlanCloneHelper.java:63)
> 2010-02-25 15:49:06,129 | ERROR | Errors | stderr     |         at
> org.apache.pig.impl.logicalLayer.LogicalPlanCloner.getClonedPlan(LogicalPlanCloner.java:45)
> 2010-02-25 15:49:06,129 | ERROR | Errors | stderr     |         at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:3083)
> 2010-02-25 15:49:06,129 | ERROR | Errors | stderr     |         at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1370)
> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr     |         at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:949)
> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr     |         at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:738)
> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr     |         at
> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr     |         at
> org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1036)
> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr     |         at
> org.apache.pig.PigServer$Graph.clone(PigServer.java:1079)
> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr     |         at
> org.apache.pig.PigServer.clonePlan(PigServer.java:390)
> 2010-02-25 15:49:06,130 | ERROR | Errors | stderr     |         at
> org.apache.pig.PigServer.compileLp(PigServer.java:806)
> 2010-02-25 15:49:06,131 | ERROR | Errors | stderr     |         at
> org.apache.pig.PigServer.compileLp(PigServer.java:793)
> 2010-02-25 15:49:06,131 | ERROR | Errors | stderr     |         at
> org.apache.pig.PigServer.store(PigServer.java:510)
>



-- 
Best Regards

Jeff Zhang