You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Bennie Schut <bs...@ebuddy.com> on 2009/11/03 17:00:50 UTC
Unable to find clone for op Const 16-169
>From time to time I receive this error:
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
to store alias uqusers11
at org.apache.pig.PigServer.store(PigServer.java:536)
at org.apache.pig.PigServer.store(PigServer.java:493)
at
com.ebuddy.dwhmapreduce.pig.chatsessions.UniqueUsers.run(UniqueUsers.java:76)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: Unable to find clone for op Const
16-169( 1 )
at
org.apache.pig.impl.logicalLayer.LogicalPlan.clone(LogicalPlan.java:139)
at
org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.<init>(LogicalPlanCloneHelper.java:63)
at
org.apache.pig.impl.logicalLayer.LogicalPlanCloner.getClonedPlan(LogicalPlanCloner.java:45)
at
org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:3044)
at
org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1328)
at
org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:907)
at
org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:696)
at
org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1034)
at org.apache.pig.PigServer$Graph.clone(PigServer.java:1077)
at org.apache.pig.PigServer.clonePlan(PigServer.java:389)
at org.apache.pig.PigServer.compileLp(PigServer.java:804)
at org.apache.pig.PigServer.compileLp(PigServer.java:791)
at org.apache.pig.PigServer.store(PigServer.java:509)
... 5 more
However running it a 2nd time it runs just fine. It's a little hard to
reproduce. I received this on a line like this:
uqusers11 = FOREACH uqusers10 GENERATE user_id, protocol, logincldr_id,
logintime_id;
but also sometimes on this:
uqusers6 = FOREACH uqusers5 GENERATE flatten($0), MIN(uqusers4.login) as
mindate;
so for now it mostly seems to happen on foreach statements (I'm
currently using pig trunk and hadoop 20.1)
Can anyone point me in the right direction on what to look at when
looking at errors like this.
Thanks,
Bennie.
Re: Unable to find clone for op Const 16-169
Posted by Vincent Barat <vi...@ubikod.com>.
Hi,
I was aware of that and then modified my code to use one PIG servers
objects for each thread.
Do you mean that even when doing this, PIG is not thread safe (not
in all its parts) ?
Ashutosh Chauhan a écrit :
> Hi Vincent,
>
> AFAIK PigServer is not thread-safe. So, it can't support queries running
> concurrently in multiple threads. As a result, you may end up in race
> conditions as the one Bennie encountered. I guess, there is a jira open to
> make PigServer thread-safe. You may want to comment there with your
> use-case.
>
> Thanks,
> Ashutosh
>
> On Tue, Nov 3, 2009 at 12:57, Vincent Barat <vi...@ubikod.com>wrote:
>
>> I have the exact same problem, and yes, I run my queries from Java
>> concurrently. The issue is easier to reproduce in local mode than in MR mode
>> (I never saw it in MR mode actually).
>>
>> I've tried to add calls to pigServer.shutdown() to see if this can help,
>> but the issue remains.
>>
>> Ashutosh Chauhan a écrit :
>>
>> Hi Bennie,
>>> Are you using Pig Java API to run your queries? If so, are you trying to
>>> run
>>> queries concurrently in multiple threads ?
>>>
>>> Ashutosh
>>>
>>> On Tue, Nov 3, 2009 at 11:00, Bennie Schut <bs...@ebuddy.com> wrote:
>>>
>>> From time to time I receive this error:
>>>> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
>>>> to store alias uqusers11
>>>> at org.apache.pig.PigServer.store(PigServer.java:536)
>>>> at org.apache.pig.PigServer.store(PigServer.java:493)
>>>> at
>>>>
>>>>
>>>> com.ebuddy.dwhmapreduce.pig.chatsessions.UniqueUsers.run(UniqueUsers.java:76)
>>>> at
>>>>
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>>
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> Caused by: java.lang.RuntimeException: Unable to find clone for op Const
>>>> 16-169( 1 )
>>>> at
>>>> org.apache.pig.impl.logicalLayer.LogicalPlan.clone(LogicalPlan.java:139)
>>>> at
>>>>
>>>>
>>>> org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.<init>(LogicalPlanCloneHelper.java:63)
>>>> at
>>>>
>>>>
>>>> org.apache.pig.impl.logicalLayer.LogicalPlanCloner.getClonedPlan(LogicalPlanCloner.java:45)
>>>> at
>>>>
>>>>
>>>> org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:3044)
>>>> at
>>>>
>>>>
>>>> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1328)
>>>> at
>>>>
>>>>
>>>> org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:907)
>>>> at
>>>>
>>>>
>>>> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:696)
>>>> at
>>>>
>>>>
>>>> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
>>>> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1034)
>>>> at org.apache.pig.PigServer$Graph.clone(PigServer.java:1077)
>>>> at org.apache.pig.PigServer.clonePlan(PigServer.java:389)
>>>> at org.apache.pig.PigServer.compileLp(PigServer.java:804)
>>>> at org.apache.pig.PigServer.compileLp(PigServer.java:791)
>>>> at org.apache.pig.PigServer.store(PigServer.java:509)
>>>> ... 5 more
>>>>
>>>> However running it a 2nd time it runs just fine. It's a little hard to
>>>> reproduce. I received this on a line like this:
>>>> uqusers11 = FOREACH uqusers10 GENERATE user_id, protocol, logincldr_id,
>>>> logintime_id;
>>>>
>>>> but also sometimes on this:
>>>> uqusers6 = FOREACH uqusers5 GENERATE flatten($0), MIN(uqusers4.login) as
>>>> mindate;
>>>>
>>>> so for now it mostly seems to happen on foreach statements (I'm
>>>> currently using pig trunk and hadoop 20.1)
>>>>
>>>> Can anyone point me in the right direction on what to look at when
>>>> looking at errors like this.
>>>> Thanks,
>>>> Bennie.
>>>>
>>>>
>
zebra adding records.
Posted by Bennie Schut <bs...@ebuddy.com>.
Anyone know of a way to add records to an existing zebra file?
I tried this:
new BasicTable.Writer(new Path(file), config);
But received this error:
java.io.IOException: ColumnGroup.Writer failed : Index meta file already
exists: newvalues/dim/users.csv.zebra/CG0/.meta
>From the code this seems correct behavior : "If path exists and contains
what look like a complete Column Group, * ColumnGroupExists exception
will be thrown. "
Re: zebra: unknown compressor none
Posted by Ashutosh Chauhan <as...@gmail.com>.
Hi Bennie,
So, you are using Zebra for its out of box serialization and
compression support. Thanks, for the explanation.
Ashutosh
On Wed, Nov 18, 2009 at 10:43, Bennie Schut <bs...@ebuddy.com> wrote:
> Hi Ashutosh,
>
> There are only 2 columns in the original file and in the zebra file and
> this is how I use it:
>
> the screenname file contains 2 fields a number and a string and is 14G
> in size, after transforming it into zebra 10G internally split into 80
> files.
> the chatsession file contains many fields both numeric and string and is
> 155M in size.
>
> register zebra-0.6.0-dev.jar;
> dim1258375560540 = load '/user/dwh/screenname_lzo_80.zebra' using
> org.apache.hadoop.zebra.pig.TableLoader('screenname_id, code');
> fact1258375560540 = load
> '/user/dwh/chatsessions/chatsessionsmap/output/chatsessions_1258534806969_0.csv'
> using PigStorage('\t') as (session_hash: chararray, email: chararray,
> refer_url: chararray, version: chararray, protocol: chararray,
> logintype: chararray, frontendversion: chararray, remote_ip: chararray,
> country: chararray, server_id, login_date, login_time, success,
> end_date, end_time, msg_sent, avg_msg_sent_size, msg_rcv,
> avg_msg_rcv_size, num_contacts, num_groups, num_sessions, secure_login,
> timeout, has_picture, screenname: chararray, useragent :chararray,
> error_code, masterlogin: chararray, unused :int);
> tmp1258375560540 = cogroup fact1258375560540 by screenname inner,
> dim1258375560540 by code outer PARALLEL 10;
> tmp12583755605401 = filter tmp1258375560540 by IsEmpty( dim1258375560540);
> tmp12583755605402 = foreach tmp12583755605401 generate
> flatten(fact1258375560540.screenname);
> tmp12583755605403 = distinct tmp12583755605402 PARALLEL 4;
> dump tmp12583755605403;
>
>
> It's basically trying to see if there are new values for screenname in
> the chatsessions file which are not in the screenname file.
> in sql it would be something like:
> select l.screenname
> from etl.chatsessions l
> left join etl.screenname sn on (sn.screenname = l.screenname)
> where sn.screenname is null;
>
> In sql the screenname_id field is a numeric field so it's only a couple
> of bytes per record but on the plain text file it's many bytes per
> record I guess that's where the whole types branch was trying to solve
> at least internally however on hdfs the input and output are still many
> bytes.
> I was looking for a way to serialize the text file so these numbers
> would only be a few bytes and then found zebra which pretty much will do
> this for you.
> My hunch was when I would reduce the size it would gain a little
> performance simple because of copy speed.
> You probably get similar result if you would manually use a
> serialization+compression however that's a lot of work.
>
> I'm still going to try and produce a zebra file with the same number of
> mappers as the original text file would cause to make sure the speed
> difference isn't caused by more work per mapper being done.
>
Re: zebra: unknown compressor none
Posted by Bennie Schut <bs...@ebuddy.com>.
Hi Ashutosh,
There are only 2 columns in the original file and in the zebra file and
this is how I use it:
the screenname file contains 2 fields a number and a string and is 14G
in size, after transforming it into zebra 10G internally split into 80
files.
the chatsession file contains many fields both numeric and string and is
155M in size.
register zebra-0.6.0-dev.jar;
dim1258375560540 = load '/user/dwh/screenname_lzo_80.zebra' using
org.apache.hadoop.zebra.pig.TableLoader('screenname_id, code');
fact1258375560540 = load
'/user/dwh/chatsessions/chatsessionsmap/output/chatsessions_1258534806969_0.csv'
using PigStorage('\t') as (session_hash: chararray, email: chararray,
refer_url: chararray, version: chararray, protocol: chararray,
logintype: chararray, frontendversion: chararray, remote_ip: chararray,
country: chararray, server_id, login_date, login_time, success,
end_date, end_time, msg_sent, avg_msg_sent_size, msg_rcv,
avg_msg_rcv_size, num_contacts, num_groups, num_sessions, secure_login,
timeout, has_picture, screenname: chararray, useragent :chararray,
error_code, masterlogin: chararray, unused :int);
tmp1258375560540 = cogroup fact1258375560540 by screenname inner,
dim1258375560540 by code outer PARALLEL 10;
tmp12583755605401 = filter tmp1258375560540 by IsEmpty( dim1258375560540);
tmp12583755605402 = foreach tmp12583755605401 generate
flatten(fact1258375560540.screenname);
tmp12583755605403 = distinct tmp12583755605402 PARALLEL 4;
dump tmp12583755605403;
It's basically trying to see if there are new values for screenname in
the chatsessions file which are not in the screenname file.
in sql it would be something like:
select l.screenname
from etl.chatsessions l
left join etl.screenname sn on (sn.screenname = l.screenname)
where sn.screenname is null;
In sql the screenname_id field is a numeric field so it's only a couple
of bytes per record but on the plain text file it's many bytes per
record I guess that's where the whole types branch was trying to solve
at least internally however on hdfs the input and output are still many
bytes.
I was looking for a way to serialize the text file so these numbers
would only be a few bytes and then found zebra which pretty much will do
this for you.
My hunch was when I would reduce the size it would gain a little
performance simple because of copy speed.
You probably get similar result if you would manually use a
serialization+compression however that's a lot of work.
I'm still going to try and produce a zebra file with the same number of
mappers as the original text file would cause to make sure the speed
difference isn't caused by more work per mapper being done.
Ashutosh Chauhan wrote:
> On Wed, Nov 18, 2009 at 08:32, Bennie Schut <bs...@ebuddy.com> wrote:
>
>
>> Using zebra this way gives me a 27% speed improvement over using plain
>>
>
> Interesting ! Can you add a bit more detail here? 27% speedup just
> because you are storing and loading your data through Zebra's table
> loader instead of using PigStorage. If so, is it because you have wide
> rows and you only are loading couple of columns out of many columns in
> your dataset?
>
> Thanks,
> Ashutosh
>
Re: zebra: unknown compressor none
Posted by Ashutosh Chauhan <as...@gmail.com>.
On Wed, Nov 18, 2009 at 08:32, Bennie Schut <bs...@ebuddy.com> wrote:
> Using zebra this way gives me a 27% speed improvement over using plain
Interesting ! Can you add a bit more detail here? 27% speedup just
because you are storing and loading your data through Zebra's table
loader instead of using PigStorage. If so, is it because you have wide
rows and you only are loading couple of columns out of many columns in
your dataset?
Thanks,
Ashutosh
RE: zebra: unknown compressor none
Posted by Yan Zhou <ya...@yahoo-inc.com>.
For Zebra unsorted table, the number of mappers are limited by the
number of tfiles per column group because the input splits are generated
based upon tfiles not within tfiles. But the improvement in the form of
block-based split within tfile is coming up any time soon
(https://issues.apache.org/jira/browse/PIG-1077).
Regarding compression, yes, you were right that zebra does not support
"none" as a compression method.
Yan
------ Forwarded Message
From: Bennie Schut <bs...@ebuddy.com>
Reply-To: "pig-user@hadoop.apache.org" <pi...@hadoop.apache.org>
Date: Wed, 18 Nov 2009 14:32:33 +0100
To: <pi...@hadoop.apache.org>
Subject: Re: zebra: unknown compressor none
Just for anyone else who might have this problem in the future.
I found a bit of a workaround. When you generate the zebra file/dir just
make sure you use something like a "order x by y parallel 20;" before
you do the store so in the zebra structure it will have 20 files and any
jobs using this file can then at least use 20 mappers.
It's not perfect though so if someone finds another way please let me
know.
Using zebra this way gives me a 27% speed improvement over using plain
text tab delimited files so even with the hack I'm happy :)
Bennie Schut wrote:
> Hi all,
>
> I still can't get pig to use multiple mappers when using zebra. I
tried
> using lzo hoping it would help but sadly no. The file is 14G tab
> delimited plain text and when using zebra with gz 7G and with lzo 10G.
> When I use the tab delimited file I get 216 mappers but with zebra
just
> 2 mappers of which 1 mapper is done almost instantly and the other
runs
> for hours. Any idea why it's not using more mappers?
>
> As an example of what I'm trying to do:
> dim1258375560540 = load '/user/dwh/screenname2.zebra' using
> org.apache.hadoop.zebra.pig.TableLoader('screenname_id, code');
> fact1258375560540 = load
> '/user/bennies/newvalues//chatsessions_1238624404177_small.csv' using
> PigStorage('\t') as (session_hash: chararray, email: chararray,
> screenname: chararray);
> tmp1258375560540 = cogroup fact1258375560540 by screenname inner,
> dim1258375560540 by code outer PARALLEL 4;
> dump tmp1258375560540;
>
>
> Thanks,
> Bennie
>
> Bennie Schut wrote:
>
>> Another zebra related question.
>>
>> I couldn't find a lot of documentation on zebra but I figured you can
>> change compression codec with a syntax like this:
>> store outfile into '/user/dwh/screenname2.zebra' using
>> org.apache.hadoop.zebra.pig.TableStorer('compress by lzo');
>>
>> And in theory disable compression like this:
>> store outfile into '/user/dwh/screenname3.zebra' using
>> org.apache.hadoop.zebra.pig.TableStorer('compress by none');
>>
>> But it doesn't seem to understand the "none" as a compressor.
>> java.io.IOException: ColumnGroup.Writer constructor failed :
Partition
>> constructor failed :Encountered " <IDENTIFIER> "none "" at line 1,
>> column 13.
>> Was
>> expecting:
>>
>> <COMPRESSOR>
>> ...
>>
>>
>>
>> at
>>
org.apache.hadoop.zebra.io.BasicTable$Writer.<init>(BasicTable.java:1116
)
>>
>> at
>>
org.apache.hadoop.zebra.pig.TableOutputFormat.checkOutputSpecs(TableStor
er.java:
154)
>>
>> at
>>
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:772)
>>
>> at
>> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>>
>> at
>> org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>>
>> at
>>
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl
.java:24
7)
>>
>> at
>>
org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
>>
>> at
>> java.lang.Thread.run(Thread.java:619)
>>
>>
>>
>> I actually tried this because when I use the zebra result on further
>> processing it only uses 2 mappers instead of the 230 mappers on the
>> original file. I remember hadoop can not split gz files so I figured
>> using compression might cause it to use so little mappers. Anyone
>> perhaps know a different approach on this?
>>
>> Thanks,
>> Bennie.
>>
>>
>>
>
>
------ End of Forwarded Message
Re: zebra: unknown compressor none
Posted by Bennie Schut <bs...@ebuddy.com>.
Just for anyone else who might have this problem in the future.
I found a bit of a workaround. When you generate the zebra file/dir just
make sure you use something like a "order x by y parallel 20;" before
you do the store so in the zebra structure it will have 20 files and any
jobs using this file can then at least use 20 mappers.
It's not perfect though so if someone finds another way please let me know.
Using zebra this way gives me a 27% speed improvement over using plain
text tab delimited files so even with the hack I'm happy :)
Bennie Schut wrote:
> Hi all,
>
> I still can't get pig to use multiple mappers when using zebra. I tried
> using lzo hoping it would help but sadly no. The file is 14G tab
> delimited plain text and when using zebra with gz 7G and with lzo 10G.
> When I use the tab delimited file I get 216 mappers but with zebra just
> 2 mappers of which 1 mapper is done almost instantly and the other runs
> for hours. Any idea why it's not using more mappers?
>
> As an example of what I'm trying to do:
> dim1258375560540 = load '/user/dwh/screenname2.zebra' using
> org.apache.hadoop.zebra.pig.TableLoader('screenname_id, code');
> fact1258375560540 = load
> '/user/bennies/newvalues//chatsessions_1238624404177_small.csv' using
> PigStorage('\t') as (session_hash: chararray, email: chararray,
> screenname: chararray);
> tmp1258375560540 = cogroup fact1258375560540 by screenname inner,
> dim1258375560540 by code outer PARALLEL 4;
> dump tmp1258375560540;
>
>
> Thanks,
> Bennie
>
> Bennie Schut wrote:
>
>> Another zebra related question.
>>
>> I couldn't find a lot of documentation on zebra but I figured you can
>> change compression codec with a syntax like this:
>> store outfile into '/user/dwh/screenname2.zebra' using
>> org.apache.hadoop.zebra.pig.TableStorer('compress by lzo');
>>
>> And in theory disable compression like this:
>> store outfile into '/user/dwh/screenname3.zebra' using
>> org.apache.hadoop.zebra.pig.TableStorer('compress by none');
>>
>> But it doesn't seem to understand the "none" as a compressor.
>> java.io.IOException: ColumnGroup.Writer constructor failed : Partition
>> constructor failed :Encountered " <IDENTIFIER> "none "" at line 1,
>> column 13.
>> Was
>> expecting:
>>
>> <COMPRESSOR>
>> ...
>>
>>
>>
>> at
>> org.apache.hadoop.zebra.io.BasicTable$Writer.<init>(BasicTable.java:1116)
>>
>> at
>> org.apache.hadoop.zebra.pig.TableOutputFormat.checkOutputSpecs(TableStorer.java:154)
>>
>> at
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:772)
>>
>> at
>> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>>
>> at
>> org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>>
>> at
>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
>>
>> at
>> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
>>
>> at
>> java.lang.Thread.run(Thread.java:619)
>>
>>
>>
>> I actually tried this because when I use the zebra result on further
>> processing it only uses 2 mappers instead of the 230 mappers on the
>> original file. I remember hadoop can not split gz files so I figured
>> using compression might cause it to use so little mappers. Anyone
>> perhaps know a different approach on this?
>>
>> Thanks,
>> Bennie.
>>
>>
>>
>
>
Re: zebra: unknown compressor none
Posted by Bennie Schut <bs...@ebuddy.com>.
Hi all,
I still can't get pig to use multiple mappers when using zebra. I tried
using lzo hoping it would help but sadly no. The file is 14G tab
delimited plain text and when using zebra with gz 7G and with lzo 10G.
When I use the tab delimited file I get 216 mappers but with zebra just
2 mappers of which 1 mapper is done almost instantly and the other runs
for hours. Any idea why it's not using more mappers?
As an example of what I'm trying to do:
dim1258375560540 = load '/user/dwh/screenname2.zebra' using
org.apache.hadoop.zebra.pig.TableLoader('screenname_id, code');
fact1258375560540 = load
'/user/bennies/newvalues//chatsessions_1238624404177_small.csv' using
PigStorage('\t') as (session_hash: chararray, email: chararray,
screenname: chararray);
tmp1258375560540 = cogroup fact1258375560540 by screenname inner,
dim1258375560540 by code outer PARALLEL 4;
dump tmp1258375560540;
Thanks,
Bennie
Bennie Schut wrote:
> Another zebra related question.
>
> I couldn't find a lot of documentation on zebra but I figured you can
> change compression codec with a syntax like this:
> store outfile into '/user/dwh/screenname2.zebra' using
> org.apache.hadoop.zebra.pig.TableStorer('compress by lzo');
>
> And in theory disable compression like this:
> store outfile into '/user/dwh/screenname3.zebra' using
> org.apache.hadoop.zebra.pig.TableStorer('compress by none');
>
> But it doesn't seem to understand the "none" as a compressor.
> java.io.IOException: ColumnGroup.Writer constructor failed : Partition
> constructor failed :Encountered " <IDENTIFIER> "none "" at line 1,
> column 13.
> Was
> expecting:
>
> <COMPRESSOR>
> ...
>
>
>
> at
> org.apache.hadoop.zebra.io.BasicTable$Writer.<init>(BasicTable.java:1116)
>
> at
> org.apache.hadoop.zebra.pig.TableOutputFormat.checkOutputSpecs(TableStorer.java:154)
>
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:772)
>
> at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>
> at
> org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
>
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
>
> at
> java.lang.Thread.run(Thread.java:619)
>
>
>
> I actually tried this because when I use the zebra result on further
> processing it only uses 2 mappers instead of the 230 mappers on the
> original file. I remember hadoop can not split gz files so I figured
> using compression might cause it to use so little mappers. Anyone
> perhaps know a different approach on this?
>
> Thanks,
> Bennie.
>
>
zebra: unknown compressor none
Posted by Bennie Schut <bs...@ebuddy.com>.
Another zebra related question.
I couldn't find a lot of documentation on zebra but I figured you can
change compression codec with a syntax like this:
store outfile into '/user/dwh/screenname2.zebra' using
org.apache.hadoop.zebra.pig.TableStorer('compress by lzo');
And in theory disable compression like this:
store outfile into '/user/dwh/screenname3.zebra' using
org.apache.hadoop.zebra.pig.TableStorer('compress by none');
But it doesn't seem to understand the "none" as a compressor.
java.io.IOException: ColumnGroup.Writer constructor failed : Partition
constructor failed :Encountered " <IDENTIFIER> "none "" at line 1,
column 13.
Was
expecting:
<COMPRESSOR>
...
at
org.apache.hadoop.zebra.io.BasicTable$Writer.<init>(BasicTable.java:1116)
at
org.apache.hadoop.zebra.pig.TableOutputFormat.checkOutputSpecs(TableStorer.java:154)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:772)
at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at
org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at
java.lang.Thread.run(Thread.java:619)
I actually tried this because when I use the zebra result on further
processing it only uses 2 mappers instead of the 230 mappers on the
original file. I remember hadoop can not split gz files so I figured
using compression might cause it to use so little mappers. Anyone
perhaps know a different approach on this?
Thanks,
Bennie.
Re: pig using zebra, ClassNotFoundException on TableOutputFormat
Posted by Bennie Schut <bs...@ebuddy.com>.
Ah thanks. Working like a charm now. Now I can play with the
TableInserter part.
Santhosh Srinivasan wrote:
> Bennie,
>
> Include zebra-0.6.0-dev.jar in your classpath and then relaunch pig.
>
> Santhosh
>
> -----Original Message-----
> From: Bennie Schut [mailto:bschut@ebuddy.com]
> Sent: Friday, November 13, 2009 3:03 AM
> To: pig-user@hadoop.apache.org
> Subject: pig using zebra, ClassNotFoundException on TableOutputFormat
>
> I'm looking into improving the performance of one of my pig jobs. I
> figured storing the data which I keep reusing in a binary/serialized
> format could help me a little with this and thus stumbled upon zebra.
> It seems like a nice abstraction and seems to do exactly what I want to
> achieve.
>
> I started with something simple but that doesn't work.
>
> register zebra-0.6.0-dev.jar;
> dim_calendar = load '/user/dwh/dim/calendar.csv' using PigStorage('\t')
> as (cldr_id: long, iso_date: chararray); outfile = order dim_calendar by
> iso_date parallel 1; store outfile into '/user/dwh/calendar.zebra' using
> org.apache.hadoop.zebra.pig.TableStorer('cldr_id: long,
> iso_date:string');
>
> On running this I get:
> ---------------
> ERROR 2117: Unexpected error when launching map reduce job.
>
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
> to store alias 97 at
> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1003)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:385)
> at
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:720)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptPar
> ser.java:324)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java
> :168)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java
> :144)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:75)
> at org.apache.pig.Main.main(Main.java:352)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
> 2117: Unexpected error when launching map reduce job.
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
> uncher.launchPig(MapReduceLauncher.java:194)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(H
> ExecutionEngine.java:249)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:780)
> at org.apache.pig.PigServer.execute(PigServer.java:773)
> at org.apache.pig.PigServer.access$100(PigServer.java:89)
> at org.apache.pig.PigServer$Graph.execute(PigServer.java:951)
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:998)
> ... 7 more
> Caused by: java.lang.RuntimeException: Could not resolve error that
> occured when launching map reduce job: java.lang.RuntimeException:
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.hadoop.zebra.pig.TableOutputFormat
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
> uncher$JobControlThreadExceptionHandler.uncaughtException(MapReduceLaunc
> her.java:428)
> at java.lang.Thread.dispatchUncaughtException(Thread.java:1831)
> -----
>
> Any idea why?
> TableOutputFormat is an inner class of TableStorer so I'm a little
> puzzled how it could find one but not the other.
> fyi.. I'm using hadoop-0.20.1 and pig/zebra from trunk but haven't
> updated pig in a few weeks.
>
> Thanks,
> Bennie.
>
RE: pig using zebra, ClassNotFoundException on TableOutputFormat
Posted by Santhosh Srinivasan <sm...@yahoo-inc.com>.
Bennie,
Include zebra-0.6.0-dev.jar in your classpath and then relaunch pig.
Santhosh
-----Original Message-----
From: Bennie Schut [mailto:bschut@ebuddy.com]
Sent: Friday, November 13, 2009 3:03 AM
To: pig-user@hadoop.apache.org
Subject: pig using zebra, ClassNotFoundException on TableOutputFormat
I'm looking into improving the performance of one of my pig jobs. I
figured storing the data which I keep reusing in a binary/serialized
format could help me a little with this and thus stumbled upon zebra.
It seems like a nice abstraction and seems to do exactly what I want to
achieve.
I started with something simple but that doesn't work.
register zebra-0.6.0-dev.jar;
dim_calendar = load '/user/dwh/dim/calendar.csv' using PigStorage('\t')
as (cldr_id: long, iso_date: chararray); outfile = order dim_calendar by
iso_date parallel 1; store outfile into '/user/dwh/calendar.zebra' using
org.apache.hadoop.zebra.pig.TableStorer('cldr_id: long,
iso_date:string');
On running this I get:
---------------
ERROR 2117: Unexpected error when launching map reduce job.
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
to store alias 97 at
org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1003)
at org.apache.pig.PigServer.registerQuery(PigServer.java:385)
at
org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:720)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptPar
ser.java:324)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java
:168)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java
:144)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:75)
at org.apache.pig.Main.main(Main.java:352)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
2117: Unexpected error when launching map reduce job.
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
uncher.launchPig(MapReduceLauncher.java:194)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(H
ExecutionEngine.java:249)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:780)
at org.apache.pig.PigServer.execute(PigServer.java:773)
at org.apache.pig.PigServer.access$100(PigServer.java:89)
at org.apache.pig.PigServer$Graph.execute(PigServer.java:951)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:998)
... 7 more
Caused by: java.lang.RuntimeException: Could not resolve error that
occured when launching map reduce job: java.lang.RuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.hadoop.zebra.pig.TableOutputFormat
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
uncher$JobControlThreadExceptionHandler.uncaughtException(MapReduceLaunc
her.java:428)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1831)
-----
Any idea why?
TableOutputFormat is an inner class of TableStorer so I'm a little
puzzled how it could find one but not the other.
fyi.. I'm using hadoop-0.20.1 and pig/zebra from trunk but haven't
updated pig in a few weeks.
Thanks,
Bennie.
pig using zebra, ClassNotFoundException on TableOutputFormat
Posted by Bennie Schut <bs...@ebuddy.com>.
I'm looking into improving the performance of one of my pig jobs. I
figured storing the data which I keep reusing in a binary/serialized
format could help me a little with this and thus stumbled upon zebra.
It seems like a nice abstraction and seems to do exactly what I want to
achieve.
I started with something simple but that doesn't work.
register zebra-0.6.0-dev.jar;
dim_calendar = load '/user/dwh/dim/calendar.csv' using PigStorage('\t')
as (cldr_id: long, iso_date: chararray);
outfile = order dim_calendar by iso_date parallel 1;
store outfile into '/user/dwh/calendar.zebra' using
org.apache.hadoop.zebra.pig.TableStorer('cldr_id: long, iso_date:string');
On running this I get:
---------------
ERROR 2117: Unexpected error when launching map reduce job.
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
to store alias 97
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1003)
at org.apache.pig.PigServer.registerQuery(PigServer.java:385)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:720)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:324)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:75)
at org.apache.pig.Main.main(Main.java:352)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
2117: Unexpected error when launching map reduce job.
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:194)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:249)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:780)
at org.apache.pig.PigServer.execute(PigServer.java:773)
at org.apache.pig.PigServer.access$100(PigServer.java:89)
at org.apache.pig.PigServer$Graph.execute(PigServer.java:951)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:998)
... 7 more
Caused by: java.lang.RuntimeException: Could not resolve error that
occured when launching map reduce job: java.lang.RuntimeException:
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.hadoop.zebra.pig.TableOutputFormat
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$JobControlThreadExceptionHandler.uncaughtException(MapReduceLauncher.java:428)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1831)
-----
Any idea why?
TableOutputFormat is an inner class of TableStorer so I'm a little
puzzled how it could find one but not the other.
fyi.. I'm using hadoop-0.20.1 and pig/zebra from trunk but haven't
updated pig in a few weeks.
Thanks,
Bennie.
Re: Unable to find clone for op Const 16-169
Posted by Bennie Schut <bs...@ebuddy.com>.
Ah thanks everyone for the help on this.
Yes Ashutosh was correct I was also using multiple threads so I will try
this patch after I get into work.
zjffdu wrote:
> Yes, Pig is not thread safe now. There's a jira item for this issue.
> https://issues.apache.org/jira/browse/PIG-240
>
> I attach a patch for this item. I am not sure whether this patch resolve
> this issue completely.
> But at least it works for me. I have ready used this patch for several
> months. Maybe you can try this patch.
>
>
> Jeff Zhang
>
>
>
> -----Original Message-----
> From: Ashutosh Chauhan [mailto:ashutosh.chauhan@gmail.com]
> Sent: 2009年11月3日 10:05
> To: pig-user@hadoop.apache.org
> Subject: Re: Unable to find clone for op Const 16-169
>
> Hi Vincent,
>
> AFAIK PigServer is not thread-safe. So, it can't support queries running
> concurrently in multiple threads. As a result, you may end up in race
> conditions as the one Bennie encountered. I guess, there is a jira open to
> make PigServer thread-safe. You may want to comment there with your
> use-case.
>
> Thanks,
> Ashutosh
>
> On Tue, Nov 3, 2009 at 12:57, Vincent Barat <vi...@ubikod.com>wrote:
>
>
>> I have the exact same problem, and yes, I run my queries from Java
>> concurrently. The issue is easier to reproduce in local mode than in MR
>>
> mode
>
>> (I never saw it in MR mode actually).
>>
>> I've tried to add calls to pigServer.shutdown() to see if this can help,
>> but the issue remains.
>>
>> Ashutosh Chauhan a écrit :
>>
>> Hi Bennie,
>>
>>> Are you using Pig Java API to run your queries? If so, are you trying to
>>> run
>>> queries concurrently in multiple threads ?
>>>
>>> Ashutosh
>>>
>>> On Tue, Nov 3, 2009 at 11:00, Bennie Schut <bs...@ebuddy.com> wrote:
>>>
>>> From time to time I receive this error:
>>>
>>>> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
>>>> to store alias uqusers11
>>>> at org.apache.pig.PigServer.store(PigServer.java:536)
>>>> at org.apache.pig.PigServer.store(PigServer.java:493)
>>>> at
>>>>
>>>>
>>>>
>>>>
> com.ebuddy.dwhmapreduce.pig.chatsessions.UniqueUsers.run(UniqueUsers.java:76
> )
>
>>>> at
>>>>
>>>>
>>>>
>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>
>>>> at
>>>>
>>>>
>>>>
>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> Caused by: java.lang.RuntimeException: Unable to find clone for op Const
>>>> 16-169( 1 )
>>>> at
>>>> org.apache.pig.impl.logicalLayer.LogicalPlan.clone(LogicalPlan.java:139)
>>>> at
>>>>
>>>>
>>>>
>>>>
> org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.<init>(LogicalPlanCl
> oneHelper.java:63)
>
>>>> at
>>>>
>>>>
>>>>
>>>>
> org.apache.pig.impl.logicalLayer.LogicalPlanCloner.getClonedPlan(LogicalPlan
> Cloner.java:45)
>
>>>> at
>>>>
>>>>
>>>>
>>>>
> org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParse
> r.java:3044)
>
>>>> at
>>>>
>>>>
>>>>
>>>>
> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.jav
> a:1328)
>
>>>> at
>>>>
>>>>
>>>>
>>>>
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:90
> 7)
>
>>>> at
>>>>
>>>>
>>>>
>>>>
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:6
> 96)
>
>>>> at
>>>>
>>>>
>>>>
>>>>
> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder
> .java:63)
>
>>>> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1034)
>>>> at org.apache.pig.PigServer$Graph.clone(PigServer.java:1077)
>>>> at org.apache.pig.PigServer.clonePlan(PigServer.java:389)
>>>> at org.apache.pig.PigServer.compileLp(PigServer.java:804)
>>>> at org.apache.pig.PigServer.compileLp(PigServer.java:791)
>>>> at org.apache.pig.PigServer.store(PigServer.java:509)
>>>> ... 5 more
>>>>
>>>> However running it a 2nd time it runs just fine. It's a little hard to
>>>> reproduce. I received this on a line like this:
>>>> uqusers11 = FOREACH uqusers10 GENERATE user_id, protocol, logincldr_id,
>>>> logintime_id;
>>>>
>>>> but also sometimes on this:
>>>> uqusers6 = FOREACH uqusers5 GENERATE flatten($0), MIN(uqusers4.login) as
>>>> mindate;
>>>>
>>>> so for now it mostly seems to happen on foreach statements (I'm
>>>> currently using pig trunk and hadoop 20.1)
>>>>
>>>> Can anyone point me in the right direction on what to look at when
>>>> looking at errors like this.
>>>> Thanks,
>>>> Bennie.
>>>>
>>>>
>>>>
>
>
Re: Unable to find clone for op Const 16-169
Posted by Jeff Zhang <zj...@gmail.com>.
Yes, even when you create a new PigServer each thread, they are still not
thread safe. Because they are in one JVM, and some static variable is
shared.
Jeff Zhang
2009/11/4 Vincent Barat <vi...@ubikod.com>
> I will try this patch too...
>
> zjffdu a écrit :
> > Yes, Pig is not thread safe now. There's a jira item for this issue.
> > https://issues.apache.org/jira/browse/PIG-240
> >
> > I attach a patch for this item. I am not sure whether this patch resolve
> > this issue completely.
> > But at least it works for me. I have ready used this patch for several
> > months. Maybe you can try this patch.
> >
> >
> > Jeff Zhang
> >
> >
> >
> > -----Original Message-----
> > From: Ashutosh Chauhan [mailto:ashutosh.chauhan@gmail.com]
> > Sent: 2009年11月3日 10:05
> > To: pig-user@hadoop.apache.org
> > Subject: Re: Unable to find clone for op Const 16-169
> >
> > Hi Vincent,
> >
> > AFAIK PigServer is not thread-safe. So, it can't support queries running
> > concurrently in multiple threads. As a result, you may end up in race
> > conditions as the one Bennie encountered. I guess, there is a jira open
> to
> > make PigServer thread-safe. You may want to comment there with your
> > use-case.
> >
> > Thanks,
> > Ashutosh
> >
> > On Tue, Nov 3, 2009 at 12:57, Vincent Barat <vincent.barat@ubikod.com
> >wrote:
> >
> >> I have the exact same problem, and yes, I run my queries from Java
> >> concurrently. The issue is easier to reproduce in local mode than in MR
> > mode
> >> (I never saw it in MR mode actually).
> >>
> >> I've tried to add calls to pigServer.shutdown() to see if this can help,
> >> but the issue remains.
> >>
> >> Ashutosh Chauhan a écrit :
> >>
> >> Hi Bennie,
> >>> Are you using Pig Java API to run your queries? If so, are you trying
> to
> >>> run
> >>> queries concurrently in multiple threads ?
> >>>
> >>> Ashutosh
> >>>
> >>> On Tue, Nov 3, 2009 at 11:00, Bennie Schut <bs...@ebuddy.com> wrote:
> >>>
> >>> From time to time I receive this error:
> >>>> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
> >>>> to store alias uqusers11
> >>>> at org.apache.pig.PigServer.store(PigServer.java:536)
> >>>> at org.apache.pig.PigServer.store(PigServer.java:493)
> >>>> at
> >>>>
> >>>>
> >>>>
> >
> com.ebuddy.dwhmapreduce.pig.chatsessions.UniqueUsers.run(UniqueUsers.java:76
> > )
> >>>> at
> >>>>
> >>>>
> >>>>
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> > va:886)
> >>>> at
> >>>>
> >>>>
> >>>>
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> > 08)
> >>>> at java.lang.Thread.run(Thread.java:619)
> >>>> Caused by: java.lang.RuntimeException: Unable to find clone for op
> Const
> >>>> 16-169( 1 )
> >>>> at
> >>>>
> org.apache.pig.impl.logicalLayer.LogicalPlan.clone(LogicalPlan.java:139)
> >>>> at
> >>>>
> >>>>
> >>>>
> >
> org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.<init>(LogicalPlanCl
> > oneHelper.java:63)
> >>>> at
> >>>>
> >>>>
> >>>>
> >
> org.apache.pig.impl.logicalLayer.LogicalPlanCloner.getClonedPlan(LogicalPlan
> > Cloner.java:45)
> >>>> at
> >>>>
> >>>>
> >>>>
> >
> org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParse
> > r.java:3044)
> >>>> at
> >>>>
> >>>>
> >>>>
> >
> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.jav
> > a:1328)
> >>>> at
> >>>>
> >>>>
> >>>>
> >
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:90
> > 7)
> >>>> at
> >>>>
> >>>>
> >>>>
> >
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:6
> > 96)
> >>>> at
> >>>>
> >>>>
> >>>>
> >
> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder
> > .java:63)
> >>>> at
> org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1034)
> >>>> at org.apache.pig.PigServer$Graph.clone(PigServer.java:1077)
> >>>> at org.apache.pig.PigServer.clonePlan(PigServer.java:389)
> >>>> at org.apache.pig.PigServer.compileLp(PigServer.java:804)
> >>>> at org.apache.pig.PigServer.compileLp(PigServer.java:791)
> >>>> at org.apache.pig.PigServer.store(PigServer.java:509)
> >>>> ... 5 more
> >>>>
> >>>> However running it a 2nd time it runs just fine. It's a little hard to
> >>>> reproduce. I received this on a line like this:
> >>>> uqusers11 = FOREACH uqusers10 GENERATE user_id, protocol,
> logincldr_id,
> >>>> logintime_id;
> >>>>
> >>>> but also sometimes on this:
> >>>> uqusers6 = FOREACH uqusers5 GENERATE flatten($0), MIN(uqusers4.login)
> as
> >>>> mindate;
> >>>>
> >>>> so for now it mostly seems to happen on foreach statements (I'm
> >>>> currently using pig trunk and hadoop 20.1)
> >>>>
> >>>> Can anyone point me in the right direction on what to look at when
> >>>> looking at errors like this.
> >>>> Thanks,
> >>>> Bennie.
> >>>>
> >>>>
> >
> >
> >
>
Re: Unable to find clone for op Const 16-169
Posted by Vincent Barat <vi...@ubikod.com>.
I will try this patch too...
zjffdu a écrit :
> Yes, Pig is not thread safe now. There's a jira item for this issue.
> https://issues.apache.org/jira/browse/PIG-240
>
> I attach a patch for this item. I am not sure whether this patch resolve
> this issue completely.
> But at least it works for me. I have ready used this patch for several
> months. Maybe you can try this patch.
>
>
> Jeff Zhang
>
>
>
> -----Original Message-----
> From: Ashutosh Chauhan [mailto:ashutosh.chauhan@gmail.com]
> Sent: 2009年11月3日 10:05
> To: pig-user@hadoop.apache.org
> Subject: Re: Unable to find clone for op Const 16-169
>
> Hi Vincent,
>
> AFAIK PigServer is not thread-safe. So, it can't support queries running
> concurrently in multiple threads. As a result, you may end up in race
> conditions as the one Bennie encountered. I guess, there is a jira open to
> make PigServer thread-safe. You may want to comment there with your
> use-case.
>
> Thanks,
> Ashutosh
>
> On Tue, Nov 3, 2009 at 12:57, Vincent Barat <vi...@ubikod.com>wrote:
>
>> I have the exact same problem, and yes, I run my queries from Java
>> concurrently. The issue is easier to reproduce in local mode than in MR
> mode
>> (I never saw it in MR mode actually).
>>
>> I've tried to add calls to pigServer.shutdown() to see if this can help,
>> but the issue remains.
>>
>> Ashutosh Chauhan a écrit :
>>
>> Hi Bennie,
>>> Are you using Pig Java API to run your queries? If so, are you trying to
>>> run
>>> queries concurrently in multiple threads ?
>>>
>>> Ashutosh
>>>
>>> On Tue, Nov 3, 2009 at 11:00, Bennie Schut <bs...@ebuddy.com> wrote:
>>>
>>> From time to time I receive this error:
>>>> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
>>>> to store alias uqusers11
>>>> at org.apache.pig.PigServer.store(PigServer.java:536)
>>>> at org.apache.pig.PigServer.store(PigServer.java:493)
>>>> at
>>>>
>>>>
>>>>
> com.ebuddy.dwhmapreduce.pig.chatsessions.UniqueUsers.run(UniqueUsers.java:76
> )
>>>> at
>>>>
>>>>
>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>>>> at
>>>>
>>>>
>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> Caused by: java.lang.RuntimeException: Unable to find clone for op Const
>>>> 16-169( 1 )
>>>> at
>>>> org.apache.pig.impl.logicalLayer.LogicalPlan.clone(LogicalPlan.java:139)
>>>> at
>>>>
>>>>
>>>>
> org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.<init>(LogicalPlanCl
> oneHelper.java:63)
>>>> at
>>>>
>>>>
>>>>
> org.apache.pig.impl.logicalLayer.LogicalPlanCloner.getClonedPlan(LogicalPlan
> Cloner.java:45)
>>>> at
>>>>
>>>>
>>>>
> org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParse
> r.java:3044)
>>>> at
>>>>
>>>>
>>>>
> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.jav
> a:1328)
>>>> at
>>>>
>>>>
>>>>
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:90
> 7)
>>>> at
>>>>
>>>>
>>>>
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:6
> 96)
>>>> at
>>>>
>>>>
>>>>
> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder
> .java:63)
>>>> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1034)
>>>> at org.apache.pig.PigServer$Graph.clone(PigServer.java:1077)
>>>> at org.apache.pig.PigServer.clonePlan(PigServer.java:389)
>>>> at org.apache.pig.PigServer.compileLp(PigServer.java:804)
>>>> at org.apache.pig.PigServer.compileLp(PigServer.java:791)
>>>> at org.apache.pig.PigServer.store(PigServer.java:509)
>>>> ... 5 more
>>>>
>>>> However running it a 2nd time it runs just fine. It's a little hard to
>>>> reproduce. I received this on a line like this:
>>>> uqusers11 = FOREACH uqusers10 GENERATE user_id, protocol, logincldr_id,
>>>> logintime_id;
>>>>
>>>> but also sometimes on this:
>>>> uqusers6 = FOREACH uqusers5 GENERATE flatten($0), MIN(uqusers4.login) as
>>>> mindate;
>>>>
>>>> so for now it mostly seems to happen on foreach statements (I'm
>>>> currently using pig trunk and hadoop 20.1)
>>>>
>>>> Can anyone point me in the right direction on what to look at when
>>>> looking at errors like this.
>>>> Thanks,
>>>> Bennie.
>>>>
>>>>
>
>
>
RE: Unable to find clone for op Const 16-169
Posted by zjffdu <zj...@gmail.com>.
Yes, Pig is not thread safe now. There's a jira item for this issue.
https://issues.apache.org/jira/browse/PIG-240
I attach a patch for this item. I am not sure whether this patch resolve
this issue completely.
But at least it works for me. I have ready used this patch for several
months. Maybe you can try this patch.
Jeff Zhang
-----Original Message-----
From: Ashutosh Chauhan [mailto:ashutosh.chauhan@gmail.com]
Sent: 2009年11月3日 10:05
To: pig-user@hadoop.apache.org
Subject: Re: Unable to find clone for op Const 16-169
Hi Vincent,
AFAIK PigServer is not thread-safe. So, it can't support queries running
concurrently in multiple threads. As a result, you may end up in race
conditions as the one Bennie encountered. I guess, there is a jira open to
make PigServer thread-safe. You may want to comment there with your
use-case.
Thanks,
Ashutosh
On Tue, Nov 3, 2009 at 12:57, Vincent Barat <vi...@ubikod.com>wrote:
> I have the exact same problem, and yes, I run my queries from Java
> concurrently. The issue is easier to reproduce in local mode than in MR
mode
> (I never saw it in MR mode actually).
>
> I've tried to add calls to pigServer.shutdown() to see if this can help,
> but the issue remains.
>
> Ashutosh Chauhan a écrit :
>
> Hi Bennie,
>>
>> Are you using Pig Java API to run your queries? If so, are you trying to
>> run
>> queries concurrently in multiple threads ?
>>
>> Ashutosh
>>
>> On Tue, Nov 3, 2009 at 11:00, Bennie Schut <bs...@ebuddy.com> wrote:
>>
>> From time to time I receive this error:
>>>
>>> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
>>> to store alias uqusers11
>>> at org.apache.pig.PigServer.store(PigServer.java:536)
>>> at org.apache.pig.PigServer.store(PigServer.java:493)
>>> at
>>>
>>>
>>>
com.ebuddy.dwhmapreduce.pig.chatsessions.UniqueUsers.run(UniqueUsers.java:76
)
>>> at
>>>
>>>
>>>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)
>>> at
>>>
>>>
>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)
>>> at java.lang.Thread.run(Thread.java:619)
>>> Caused by: java.lang.RuntimeException: Unable to find clone for op Const
>>> 16-169( 1 )
>>> at
>>> org.apache.pig.impl.logicalLayer.LogicalPlan.clone(LogicalPlan.java:139)
>>> at
>>>
>>>
>>>
org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.<init>(LogicalPlanCl
oneHelper.java:63)
>>> at
>>>
>>>
>>>
org.apache.pig.impl.logicalLayer.LogicalPlanCloner.getClonedPlan(LogicalPlan
Cloner.java:45)
>>> at
>>>
>>>
>>>
org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParse
r.java:3044)
>>> at
>>>
>>>
>>>
org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.jav
a:1328)
>>> at
>>>
>>>
>>>
org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:90
7)
>>> at
>>>
>>>
>>>
org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:6
96)
>>> at
>>>
>>>
>>>
org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder
.java:63)
>>> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1034)
>>> at org.apache.pig.PigServer$Graph.clone(PigServer.java:1077)
>>> at org.apache.pig.PigServer.clonePlan(PigServer.java:389)
>>> at org.apache.pig.PigServer.compileLp(PigServer.java:804)
>>> at org.apache.pig.PigServer.compileLp(PigServer.java:791)
>>> at org.apache.pig.PigServer.store(PigServer.java:509)
>>> ... 5 more
>>>
>>> However running it a 2nd time it runs just fine. It's a little hard to
>>> reproduce. I received this on a line like this:
>>> uqusers11 = FOREACH uqusers10 GENERATE user_id, protocol, logincldr_id,
>>> logintime_id;
>>>
>>> but also sometimes on this:
>>> uqusers6 = FOREACH uqusers5 GENERATE flatten($0), MIN(uqusers4.login) as
>>> mindate;
>>>
>>> so for now it mostly seems to happen on foreach statements (I'm
>>> currently using pig trunk and hadoop 20.1)
>>>
>>> Can anyone point me in the right direction on what to look at when
>>> looking at errors like this.
>>> Thanks,
>>> Bennie.
>>>
>>>
>>
Re: Unable to find clone for op Const 16-169
Posted by Ashutosh Chauhan <as...@gmail.com>.
Hi Vincent,
AFAIK PigServer is not thread-safe. So, it can't support queries running
concurrently in multiple threads. As a result, you may end up in race
conditions as the one Bennie encountered. I guess, there is a jira open to
make PigServer thread-safe. You may want to comment there with your
use-case.
Thanks,
Ashutosh
On Tue, Nov 3, 2009 at 12:57, Vincent Barat <vi...@ubikod.com>wrote:
> I have the exact same problem, and yes, I run my queries from Java
> concurrently. The issue is easier to reproduce in local mode than in MR mode
> (I never saw it in MR mode actually).
>
> I've tried to add calls to pigServer.shutdown() to see if this can help,
> but the issue remains.
>
> Ashutosh Chauhan a écrit :
>
> Hi Bennie,
>>
>> Are you using Pig Java API to run your queries? If so, are you trying to
>> run
>> queries concurrently in multiple threads ?
>>
>> Ashutosh
>>
>> On Tue, Nov 3, 2009 at 11:00, Bennie Schut <bs...@ebuddy.com> wrote:
>>
>> From time to time I receive this error:
>>>
>>> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
>>> to store alias uqusers11
>>> at org.apache.pig.PigServer.store(PigServer.java:536)
>>> at org.apache.pig.PigServer.store(PigServer.java:493)
>>> at
>>>
>>>
>>> com.ebuddy.dwhmapreduce.pig.chatsessions.UniqueUsers.run(UniqueUsers.java:76)
>>> at
>>>
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>>
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:619)
>>> Caused by: java.lang.RuntimeException: Unable to find clone for op Const
>>> 16-169( 1 )
>>> at
>>> org.apache.pig.impl.logicalLayer.LogicalPlan.clone(LogicalPlan.java:139)
>>> at
>>>
>>>
>>> org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.<init>(LogicalPlanCloneHelper.java:63)
>>> at
>>>
>>>
>>> org.apache.pig.impl.logicalLayer.LogicalPlanCloner.getClonedPlan(LogicalPlanCloner.java:45)
>>> at
>>>
>>>
>>> org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:3044)
>>> at
>>>
>>>
>>> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1328)
>>> at
>>>
>>>
>>> org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:907)
>>> at
>>>
>>>
>>> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:696)
>>> at
>>>
>>>
>>> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
>>> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1034)
>>> at org.apache.pig.PigServer$Graph.clone(PigServer.java:1077)
>>> at org.apache.pig.PigServer.clonePlan(PigServer.java:389)
>>> at org.apache.pig.PigServer.compileLp(PigServer.java:804)
>>> at org.apache.pig.PigServer.compileLp(PigServer.java:791)
>>> at org.apache.pig.PigServer.store(PigServer.java:509)
>>> ... 5 more
>>>
>>> However running it a 2nd time it runs just fine. It's a little hard to
>>> reproduce. I received this on a line like this:
>>> uqusers11 = FOREACH uqusers10 GENERATE user_id, protocol, logincldr_id,
>>> logintime_id;
>>>
>>> but also sometimes on this:
>>> uqusers6 = FOREACH uqusers5 GENERATE flatten($0), MIN(uqusers4.login) as
>>> mindate;
>>>
>>> so for now it mostly seems to happen on foreach statements (I'm
>>> currently using pig trunk and hadoop 20.1)
>>>
>>> Can anyone point me in the right direction on what to look at when
>>> looking at errors like this.
>>> Thanks,
>>> Bennie.
>>>
>>>
>>
Re: Unable to find clone for op Const 16-169
Posted by Vincent Barat <vi...@ubikod.com>.
I have the exact same problem, and yes, I run my queries from Java
concurrently. The issue is easier to reproduce in local mode than in
MR mode (I never saw it in MR mode actually).
I've tried to add calls to pigServer.shutdown() to see if this can
help, but the issue remains.
Ashutosh Chauhan a écrit :
> Hi Bennie,
>
> Are you using Pig Java API to run your queries? If so, are you trying to run
> queries concurrently in multiple threads ?
>
> Ashutosh
>
> On Tue, Nov 3, 2009 at 11:00, Bennie Schut <bs...@ebuddy.com> wrote:
>
>> From time to time I receive this error:
>>
>> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
>> to store alias uqusers11
>> at org.apache.pig.PigServer.store(PigServer.java:536)
>> at org.apache.pig.PigServer.store(PigServer.java:493)
>> at
>>
>> com.ebuddy.dwhmapreduce.pig.chatsessions.UniqueUsers.run(UniqueUsers.java:76)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.lang.RuntimeException: Unable to find clone for op Const
>> 16-169( 1 )
>> at
>> org.apache.pig.impl.logicalLayer.LogicalPlan.clone(LogicalPlan.java:139)
>> at
>>
>> org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.<init>(LogicalPlanCloneHelper.java:63)
>> at
>>
>> org.apache.pig.impl.logicalLayer.LogicalPlanCloner.getClonedPlan(LogicalPlanCloner.java:45)
>> at
>>
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:3044)
>> at
>>
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1328)
>> at
>>
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:907)
>> at
>>
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:696)
>> at
>>
>> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
>> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1034)
>> at org.apache.pig.PigServer$Graph.clone(PigServer.java:1077)
>> at org.apache.pig.PigServer.clonePlan(PigServer.java:389)
>> at org.apache.pig.PigServer.compileLp(PigServer.java:804)
>> at org.apache.pig.PigServer.compileLp(PigServer.java:791)
>> at org.apache.pig.PigServer.store(PigServer.java:509)
>> ... 5 more
>>
>> However running it a 2nd time it runs just fine. It's a little hard to
>> reproduce. I received this on a line like this:
>> uqusers11 = FOREACH uqusers10 GENERATE user_id, protocol, logincldr_id,
>> logintime_id;
>>
>> but also sometimes on this:
>> uqusers6 = FOREACH uqusers5 GENERATE flatten($0), MIN(uqusers4.login) as
>> mindate;
>>
>> so for now it mostly seems to happen on foreach statements (I'm
>> currently using pig trunk and hadoop 20.1)
>>
>> Can anyone point me in the right direction on what to look at when
>> looking at errors like this.
>> Thanks,
>> Bennie.
>>
>
Re: Unable to find clone for op Const 16-169
Posted by Ashutosh Chauhan <as...@gmail.com>.
Hi Bennie,
Are you using Pig Java API to run your queries? If so, are you trying to run
queries concurrently in multiple threads ?
Ashutosh
On Tue, Nov 3, 2009 at 11:00, Bennie Schut <bs...@ebuddy.com> wrote:
> From time to time I receive this error:
>
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
> to store alias uqusers11
> at org.apache.pig.PigServer.store(PigServer.java:536)
> at org.apache.pig.PigServer.store(PigServer.java:493)
> at
>
> com.ebuddy.dwhmapreduce.pig.chatsessions.UniqueUsers.run(UniqueUsers.java:76)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.RuntimeException: Unable to find clone for op Const
> 16-169( 1 )
> at
> org.apache.pig.impl.logicalLayer.LogicalPlan.clone(LogicalPlan.java:139)
> at
>
> org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.<init>(LogicalPlanCloneHelper.java:63)
> at
>
> org.apache.pig.impl.logicalLayer.LogicalPlanCloner.getClonedPlan(LogicalPlanCloner.java:45)
> at
>
> org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:3044)
> at
>
> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1328)
> at
>
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:907)
> at
>
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:696)
> at
>
> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1034)
> at org.apache.pig.PigServer$Graph.clone(PigServer.java:1077)
> at org.apache.pig.PigServer.clonePlan(PigServer.java:389)
> at org.apache.pig.PigServer.compileLp(PigServer.java:804)
> at org.apache.pig.PigServer.compileLp(PigServer.java:791)
> at org.apache.pig.PigServer.store(PigServer.java:509)
> ... 5 more
>
> However running it a 2nd time it runs just fine. It's a little hard to
> reproduce. I received this on a line like this:
> uqusers11 = FOREACH uqusers10 GENERATE user_id, protocol, logincldr_id,
> logintime_id;
>
> but also sometimes on this:
> uqusers6 = FOREACH uqusers5 GENERATE flatten($0), MIN(uqusers4.login) as
> mindate;
>
> so for now it mostly seems to happen on foreach statements (I'm
> currently using pig trunk and hadoop 20.1)
>
> Can anyone point me in the right direction on what to look at when
> looking at errors like this.
> Thanks,
> Bennie.
>