You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Christopher Olston (JIRA)" <ji...@apache.org> on 2008/12/02 01:51:44 UTC
[jira] Updated: (PIG-552) UDF defined with argument causes class
instantiation exception, when UDF followed by sort
[ https://issues.apache.org/jira/browse/PIG-552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christopher Olston updated PIG-552:
-----------------------------------
Description:
I'm doing:
define myFunc myFunc('blah');
b = foreach a generate myFunc(*);
c = order b by $0;
Pig parses it, but fails when it tries to run it on hadoop (I'm using "local" mode). It tries to invoke the class loader on "myFunc('blah')" instead of on "myFunc", which causes an exception.
The bug seems to stem from this part of JobControlCompiler.getJobConf():
if(mro.UDFs.size()==1){
String compFuncSpec = mro.UDFs.get(0);
Class comparator = PigContext.resolveClassName(compFuncSpec);
if(ComparisonFunc.class.isAssignableFrom(comparator)) {
jobConf.setMapperClass(PigMapReduce.MapWithComparator.class);
jobConf.setReducerClass(PigMapReduce.ReduceWithComparator.class);
jobConf.set("pig.reduce.package", ObjectSerializer.serialize(pack));
jobConf.set("pig.usercomparator", "true");
jobConf.setOutputKeyClass(NullableTuple.class);
jobConf.setOutputKeyComparatorClass(comparator);
}
} else {
jobConf.set("pig.sortOrder",
ObjectSerializer.serialize(mro.getSortOrder()));
}
was:
I'm doing:
define myFunc myFunc('blah');
b = foreach a generate myFunc(*);
Pig parses it, but fails when it tries to run it on hadoop (I'm using "local" mode). It tries to invoke the class loader on "myFunc('blah')" instead of on "myFunc", which causes an exception.
The bug seems to stem from this part of JobControlCompiler.getJobConf():
if(mro.UDFs.size()==1){
String compFuncSpec = mro.UDFs.get(0);
Class comparator = PigContext.resolveClassName(compFuncSpec);
if(ComparisonFunc.class.isAssignableFrom(comparator)) {
jobConf.setMapperClass(PigMapReduce.MapWithComparator.class);
jobConf.setReducerClass(PigMapReduce.ReduceWithComparator.class);
jobConf.set("pig.reduce.package", ObjectSerializer.serialize(pack));
jobConf.set("pig.usercomparator", "true");
jobConf.setOutputKeyClass(NullableTuple.class);
jobConf.setOutputKeyComparatorClass(comparator);
}
} else {
jobConf.set("pig.sortOrder",
ObjectSerializer.serialize(mro.getSortOrder()));
}
Summary: UDF defined with argument causes class instantiation exception, when UDF followed by sort (was: UDF defined with argument causes class instantiation exception)
> UDF defined with argument causes class instantiation exception, when UDF followed by sort
> -----------------------------------------------------------------------------------------
>
> Key: PIG-552
> URL: https://issues.apache.org/jira/browse/PIG-552
> Project: Pig
> Issue Type: Bug
> Affects Versions: types_branch
> Reporter: Christopher Olston
>
> I'm doing:
> define myFunc myFunc('blah');
> b = foreach a generate myFunc(*);
> c = order b by $0;
> Pig parses it, but fails when it tries to run it on hadoop (I'm using "local" mode). It tries to invoke the class loader on "myFunc('blah')" instead of on "myFunc", which causes an exception.
> The bug seems to stem from this part of JobControlCompiler.getJobConf():
> if(mro.UDFs.size()==1){
> String compFuncSpec = mro.UDFs.get(0);
> Class comparator = PigContext.resolveClassName(compFuncSpec);
> if(ComparisonFunc.class.isAssignableFrom(comparator)) {
> jobConf.setMapperClass(PigMapReduce.MapWithComparator.class);
> jobConf.setReducerClass(PigMapReduce.ReduceWithComparator.class);
> jobConf.set("pig.reduce.package", ObjectSerializer.serialize(pack));
> jobConf.set("pig.usercomparator", "true");
> jobConf.setOutputKeyClass(NullableTuple.class);
> jobConf.setOutputKeyComparatorClass(comparator);
> }
> } else {
> jobConf.set("pig.sortOrder",
> ObjectSerializer.serialize(mro.getSortOrder()));
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.