You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Jonathan Coveney <jc...@gmail.com> on 2011/01/15 00:06:19 UTC
Error during parsing. could not instantiate 'squeal.map.makecountalgq'
with arguments 'null'
I did a search online, and while someone had the same error, I don't think
it was related. From the error log, I see this...
Caused by: java.lang.RuntimeException: Final function of
squeal.map.makecountalgq is not of the expected type.
at org.apache.pig.EvalFunc.<init>(EvalFunc.java:146)
at squeal.map.makecountalgq.<init>(makecountalgq.java:32)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:472)
But here is my udf (I've truncated a bit, but it works if it isn't
algebraic)
public class makecountalgq extends EvalFunc<Map<String,Integer>> implements
Algebraic, Accumulator<Map<String,Integer>> {
static TupleFactory mTupleFactory = TupleFactory.getInstance();
static BagFactory mBagFactory = BagFactory.getInstance();
private MapWrapper ourWrap = new MapWrapper();
public String getInitial() { return Initial.class.getName(); }
public String getIntermed() { return Intermed.class.getName(); }
public String getFinal() { return Final.class.getName(); }
//An initial tuple, crates partial results
public static class Initial extends EvalFunc<Tuple> {
public Tuple exec(Tuple input) throws IOException {
MapWrapper aWrap = new MapWrapper(new
HashMap<String,MutableInt>());
DataBag bag = (DataBag)input.get(0);
aWrap.addSBag(bag);
return mTupleFactory.newTuple(aWrap);
}
}
//Called on the result of the Initial, or on another intermediate
public static class Intermed extends EvalFunc<Tuple> {
public Tuple exec(Tuple input) throws IOException {
return
mTupleFactory.newTuple(mergeWrappers((DataBag)input.get(0)));
}
}
//Can take the tuple that initial or intermed creates
public static class Final extends EvalFunc<Map<String,Integer>> {
public Map<String,Integer> exec(Tuple input) throws
IOException {
return
mergeWrappers((DataBag)input.get(0)).convertMap();
}
}
//takes a bag of MapWrappers and merges them
public static MapWrapper mergeWrappers(DataBag bag) throws
IOException {
Iterator<Tuple> bagIt = bag.iterator();
MapWrapper aWrap = (MapWrapper)bagIt.next().get(0);
while (bagIt.hasNext())
aWrap.merge((MapWrapper)bagIt.next().get(0));
return aWrap;
}
}
Any idea why I would get this error? It compiles fine, and if I even just
take out Algebraic, I can use the UDF (it just uses the accumulator). I
don't see how the type is off...
Any ideas? (in case the context matters: this is a script that takes a list
of (a, b:chararray), and gives you, for a given key, a map from b-># of
times b occured.
Here is how I invoke it:
register squeal.jar;
A = LOAD 'test.txt' AS (a:chararray, b:chararray);
B = GROUP A BY a;
C = FOREACH B GENERATE group, squeal.map.makecountalgq(A.b) as mapz;
And I get the error.
Thanks in advance. I guess I could content myself with an Accumulator but
the Algebraic solution is so close... a MapWrapper is a class I created that
lets me pass around Maps and whatnot.
Re: Error during parsing. could not instantiate 'squeal.map.makecountalgq'
with arguments 'null'
Posted by Jonathan Coveney <jc...@gmail.com>.
Hmm, in that case it allows me to run it, but dies and gives me this error
(from log)
Pig Stack Trace
---------------
ERROR 1066: Unable to open iterator for alias C
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
open iterator for alias C
at org.apache.pig.PigServer.openIterator(PigServer.java:754)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
at org.apache.pig.Main.run(Main.java:465)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
at org.apache.pig.PigServer.openIterator(PigServer.java:744)
... 7 more
================================================================================
and here is what grunt gave
java.lang.RuntimeException: Unexpected data type -1 found in stream.
at
org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:478)
at
org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:541)
at
org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:357)
at
org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:541)
at
org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:357)
at org.apache.pig.data.BinSedesTuple.write(BinSedesTuple.java:57)
at
org.apache.pig.impl.io.PigNullableWritable.write(PigNullableWritable.java:123)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:900)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:541)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:116)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
not sure what this -1 business is...
2011/1/14 Dmitriy Ryaboy <dv...@gmail.com>
> Much as it pains me -- what happens if you ungenerify (just use Map instead
> of Map<String, Integer>)?
>
> On Fri, Jan 14, 2011 at 3:06 PM, Jonathan Coveney <jcoveney@gmail.com
> >wrote:
>
> > I did a search online, and while someone had the same error, I don't
> think
> > it was related. From the error log, I see this...
> >
> > Caused by: java.lang.RuntimeException: Final function of
> > squeal.map.makecountalgq is not of the expected type.
> > at org.apache.pig.EvalFunc.<init>(EvalFunc.java:146)
> > at squeal.map.makecountalgq.<init>(makecountalgq.java:32)
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> > at
> >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > at java.lang.Class.newInstance0(Class.java:355)
> > at java.lang.Class.newInstance(Class.java:308)
> > at
> >
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:472)
> >
> > But here is my udf (I've truncated a bit, but it works if it isn't
> > algebraic)
> >
> > public class makecountalgq extends EvalFunc<Map<String,Integer>>
> implements
> > Algebraic, Accumulator<Map<String,Integer>> {
> > static TupleFactory mTupleFactory = TupleFactory.getInstance();
> > static BagFactory mBagFactory = BagFactory.getInstance();
> > private MapWrapper ourWrap = new MapWrapper();
> >
> > public String getInitial() { return Initial.class.getName(); }
> >
> > public String getIntermed() { return Intermed.class.getName(); }
> >
> > public String getFinal() { return Final.class.getName(); }
> >
> > //An initial tuple, crates partial results
> > public static class Initial extends EvalFunc<Tuple> {
> > public Tuple exec(Tuple input) throws IOException {
> > MapWrapper aWrap = new MapWrapper(new
> > HashMap<String,MutableInt>());
> > DataBag bag = (DataBag)input.get(0);
> > aWrap.addSBag(bag);
> > return mTupleFactory.newTuple(aWrap);
> > }
> > }
> > //Called on the result of the Initial, or on another intermediate
> > public static class Intermed extends EvalFunc<Tuple> {
> > public Tuple exec(Tuple input) throws IOException {
> > return
> > mTupleFactory.newTuple(mergeWrappers((DataBag)input.get(0)));
> > }
> > }
> > //Can take the tuple that initial or intermed creates
> > public static class Final extends EvalFunc<Map<String,Integer>> {
> > public Map<String,Integer> exec(Tuple input) throws
> > IOException {
> > return
> > mergeWrappers((DataBag)input.get(0)).convertMap();
> > }
> > }
> >
> > //takes a bag of MapWrappers and merges them
> > public static MapWrapper mergeWrappers(DataBag bag) throws
> > IOException {
> > Iterator<Tuple> bagIt = bag.iterator();
> > MapWrapper aWrap = (MapWrapper)bagIt.next().get(0);
> > while (bagIt.hasNext())
> > aWrap.merge((MapWrapper)bagIt.next().get(0));
> > return aWrap;
> > }
> > }
> >
> > Any idea why I would get this error? It compiles fine, and if I even just
> > take out Algebraic, I can use the UDF (it just uses the accumulator). I
> > don't see how the type is off...
> >
> > Any ideas? (in case the context matters: this is a script that takes a
> list
> > of (a, b:chararray), and gives you, for a given key, a map from b-># of
> > times b occured.
> >
> > Here is how I invoke it:
> >
> > register squeal.jar;
> > A = LOAD 'test.txt' AS (a:chararray, b:chararray);
> > B = GROUP A BY a;
> > C = FOREACH B GENERATE group, squeal.map.makecountalgq(A.b) as mapz;
> >
> > And I get the error.
> >
> > Thanks in advance. I guess I could content myself with an Accumulator but
> > the Algebraic solution is so close... a MapWrapper is a class I created
> > that
> > lets me pass around Maps and whatnot.
> >
>
Re: Error during parsing. could not instantiate 'squeal.map.makecountalgq'
with arguments 'null'
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Much as it pains me -- what happens if you ungenerify (just use Map instead
of Map<String, Integer>)?
On Fri, Jan 14, 2011 at 3:06 PM, Jonathan Coveney <jc...@gmail.com>wrote:
> I did a search online, and while someone had the same error, I don't think
> it was related. From the error log, I see this...
>
> Caused by: java.lang.RuntimeException: Final function of
> squeal.map.makecountalgq is not of the expected type.
> at org.apache.pig.EvalFunc.<init>(EvalFunc.java:146)
> at squeal.map.makecountalgq.<init>(makecountalgq.java:32)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at java.lang.Class.newInstance0(Class.java:355)
> at java.lang.Class.newInstance(Class.java:308)
> at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:472)
>
> But here is my udf (I've truncated a bit, but it works if it isn't
> algebraic)
>
> public class makecountalgq extends EvalFunc<Map<String,Integer>> implements
> Algebraic, Accumulator<Map<String,Integer>> {
> static TupleFactory mTupleFactory = TupleFactory.getInstance();
> static BagFactory mBagFactory = BagFactory.getInstance();
> private MapWrapper ourWrap = new MapWrapper();
>
> public String getInitial() { return Initial.class.getName(); }
>
> public String getIntermed() { return Intermed.class.getName(); }
>
> public String getFinal() { return Final.class.getName(); }
>
> //An initial tuple, crates partial results
> public static class Initial extends EvalFunc<Tuple> {
> public Tuple exec(Tuple input) throws IOException {
> MapWrapper aWrap = new MapWrapper(new
> HashMap<String,MutableInt>());
> DataBag bag = (DataBag)input.get(0);
> aWrap.addSBag(bag);
> return mTupleFactory.newTuple(aWrap);
> }
> }
> //Called on the result of the Initial, or on another intermediate
> public static class Intermed extends EvalFunc<Tuple> {
> public Tuple exec(Tuple input) throws IOException {
> return
> mTupleFactory.newTuple(mergeWrappers((DataBag)input.get(0)));
> }
> }
> //Can take the tuple that initial or intermed creates
> public static class Final extends EvalFunc<Map<String,Integer>> {
> public Map<String,Integer> exec(Tuple input) throws
> IOException {
> return
> mergeWrappers((DataBag)input.get(0)).convertMap();
> }
> }
>
> //takes a bag of MapWrappers and merges them
> public static MapWrapper mergeWrappers(DataBag bag) throws
> IOException {
> Iterator<Tuple> bagIt = bag.iterator();
> MapWrapper aWrap = (MapWrapper)bagIt.next().get(0);
> while (bagIt.hasNext())
> aWrap.merge((MapWrapper)bagIt.next().get(0));
> return aWrap;
> }
> }
>
> Any idea why I would get this error? It compiles fine, and if I even just
> take out Algebraic, I can use the UDF (it just uses the accumulator). I
> don't see how the type is off...
>
> Any ideas? (in case the context matters: this is a script that takes a list
> of (a, b:chararray), and gives you, for a given key, a map from b-># of
> times b occured.
>
> Here is how I invoke it:
>
> register squeal.jar;
> A = LOAD 'test.txt' AS (a:chararray, b:chararray);
> B = GROUP A BY a;
> C = FOREACH B GENERATE group, squeal.map.makecountalgq(A.b) as mapz;
>
> And I get the error.
>
> Thanks in advance. I guess I could content myself with an Accumulator but
> the Algebraic solution is so close... a MapWrapper is a class I created
> that
> lets me pass around Maps and whatnot.
>