You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by da...@ya.ru on 2009/01/10 18:28:21 UTC
Bag items are casted to unexpected types?
I often meet errors with chararray bag items. It seems that a bag
item can be casted to some other type rather than specified chararray
type. May be it's just before becoming a true chararray value. But
it can produce strange errors.
I suppose that there is a try to recognize bag item type somewhere in
deserializer, right? So why the user specified type is not used
directly. And what are the symbols that a string should not have to
be not casted to other type?
The latest issue with bags:
a = load 'a' as (word: chararray, length: long, phrases: bag{t:
tuple(id: chararray)});
b = order a by word;
store b into 'b';
It gives lots of errors like:
2009-01-10 20:01:49,507 [main] ERROR
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher
- Error message from task (map)
task_200901101933_0007_m_000077java.lang.RuntimeException: Unexpected
data type 116 found in stream.
at org.apache.pig.data.DataReaderWriter.readDatum(DataReaderWriter.java:117)
at org.apache.pig.builtin.BinStorage.getNext(BinStorage.java:90)
at org.apache.pig.impl.builtin.RandomSampleLoader.getNext(RandomSampleLoader.java:44)
at org.apache.pig.backend.executionengine.PigSlice.next(PigSlice.java:101)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SliceWrapper$1.next(SliceWrapper.java:157)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SliceWrapper$1.next(SliceWrapper.java:133)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
The number in "Unexpected data type 116 found in stream" message varies.
2009-01-10 20:01:49,507 [main] ERROR
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher
- Error message from task (map)
task_200901101933_0007_m_000078java.lang.OutOfMemoryError: Java heap
space
at org.apache.pig.data.DataReaderWriter.readDatum(DataReaderWriter.java:105)
at org.apache.pig.builtin.BinStorage.getNext(BinStorage.java:90)
at org.apache.pig.impl.builtin.RandomSampleLoader.getNext(RandomSampleLoader.java:44)
at org.apache.pig.backend.executionengine.PigSlice.next(PigSlice.java:101)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SliceWrapper$1.next(SliceWrapper.java:157)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SliceWrapper$1.next(SliceWrapper.java:133)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
The data files are not very large and mapred.child.java.opts options
is -Xmx2048m.
If column 'phrases' is filtered out before ordering, everything is ok.
What is wrong with my bags usage?
Thanks.
RE: Bag items are casted to unexpected types?
Posted by Santhosh Srinivasan <sm...@yahoo-inc.com>.
This is a bug. I have created a JIRA - Pig-616
(https://issues.apache.org/jira/browse/PIG-616) to track this issue.
Summary: The data parser identifies the stream as a bag of integer/long
as the case may be. While the user has explicitly specified the bag to
be a bag of chararray, the bag of integer/long is not converted to a bag
of chararray.
Thanks for your patience.
Thanks,
Santhosh
-----Original Message-----
From: radgara@gmail.com [mailto:radgara@gmail.com] On Behalf Of
daga@ya.ru
Sent: Sunday, January 11, 2009 3:47 AM
To: pig-user@hadoop.apache.org
Subject: Re: Bag items are casted to unexpected types?
Chararray bag items definitely can be casted to another type(s).
There are details below.
How to operate bags of strings with confidence of right items type?
Script:
a = load 'a' as (list: bag{t: tuple(item: chararray)});
b = foreach a generate TransformBag(list);
UDF:
DataBag bag = (DataBag)input.get(0);
DataBag output = mBF.newDefaultBag();
for (Tuple t: bag)
output.add(mTF.newTuple(DoSomeStringTransformation((String)t.get(0))));
return output;
Data:
{(270970443)}
or any other string data that occasionally look like numbers.
Error:
java.io.IOException: Received Error while processing the reduce plan:
Caught error from UDF TransformBag[java.lang.Long cannot be cast to
java.lang.String [java.lang.Long cannot be cast to java.lang.String]]
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduc
e$Reduce.runPipeline(PigMapReduce.java:307)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduc
e$Reduce.processOnePackageOutput(PigMapReduce.java:247)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduc
e$Reduce.reduce(PigMapReduce.java:224)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduc
e$Reduce.reduce(PigMapReduce.java:136)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
2009/1/10 <da...@ya.ru>:
> I often meet errors with chararray bag items. It seems that a bag
> item can be casted to some other type rather than specified chararray
> type. May be it's just before becoming a true chararray value. But
> it can produce strange errors.
>
> I suppose that there is a try to recognize bag item type somewhere in
> deserializer, right? So why the user specified type is not used
> directly. And what are the symbols that a string should not have to
> be not casted to other type?
>
> The latest issue with bags:
>
> a = load 'a' as (word: chararray, length: long, phrases: bag{t:
> tuple(id: chararray)});
> b = order a by word;
> store b into 'b';
>
> It gives lots of errors like:
>
> 2009-01-10 20:01:49,507 [main] ERROR
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher
> - Error message from task (map)
> task_200901101933_0007_m_000077java.lang.RuntimeException: Unexpected
> data type 116 found in stream.
> at
org.apache.pig.data.DataReaderWriter.readDatum(DataReaderWriter.java:117
)
> at
org.apache.pig.builtin.BinStorage.getNext(BinStorage.java:90)
> at
org.apache.pig.impl.builtin.RandomSampleLoader.getNext(RandomSampleLoade
r.java:44)
> at
org.apache.pig.backend.executionengine.PigSlice.next(PigSlice.java:101)
> at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SliceWrappe
r$1.next(SliceWrapper.java:157)
> at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SliceWrappe
r$1.next(SliceWrapper.java:133)
> at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.
java:186)
> at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:1
70)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.Child.main(Child.java:155)
>
> The number in "Unexpected data type 116 found in stream" message
varies.
>
> 2009-01-10 20:01:49,507 [main] ERROR
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher
> - Error message from task (map)
> task_200901101933_0007_m_000078java.lang.OutOfMemoryError: Java heap
> space
> at
org.apache.pig.data.DataReaderWriter.readDatum(DataReaderWriter.java:105
)
> at
org.apache.pig.builtin.BinStorage.getNext(BinStorage.java:90)
> at
org.apache.pig.impl.builtin.RandomSampleLoader.getNext(RandomSampleLoade
r.java:44)
> at
org.apache.pig.backend.executionengine.PigSlice.next(PigSlice.java:101)
> at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SliceWrappe
r$1.next(SliceWrapper.java:157)
> at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SliceWrappe
r$1.next(SliceWrapper.java:133)
> at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.
java:186)
> at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:1
70)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.Child.main(Child.java:155)
>
> The data files are not very large and mapred.child.java.opts options
> is -Xmx2048m.
>
> If column 'phrases' is filtered out before ordering, everything is ok.
>
> What is wrong with my bags usage?
>
> Thanks.
>
Re: Bag items are casted to unexpected types?
Posted by da...@ya.ru.
Chararray bag items definitely can be casted to another type(s).
There are details below.
How to operate bags of strings with confidence of right items type?
Script:
a = load 'a' as (list: bag{t: tuple(item: chararray)});
b = foreach a generate TransformBag(list);
UDF:
DataBag bag = (DataBag)input.get(0);
DataBag output = mBF.newDefaultBag();
for (Tuple t: bag)
output.add(mTF.newTuple(DoSomeStringTransformation((String)t.get(0))));
return output;
Data:
{(270970443)}
or any other string data that occasionally look like numbers.
Error:
java.io.IOException: Received Error while processing the reduce plan:
Caught error from UDF TransformBag[java.lang.Long cannot be cast to
java.lang.String [java.lang.Long cannot be cast to java.lang.String]]
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:307)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:247)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:224)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:136)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
2009/1/10 <da...@ya.ru>:
> I often meet errors with chararray bag items. It seems that a bag
> item can be casted to some other type rather than specified chararray
> type. May be it's just before becoming a true chararray value. But
> it can produce strange errors.
>
> I suppose that there is a try to recognize bag item type somewhere in
> deserializer, right? So why the user specified type is not used
> directly. And what are the symbols that a string should not have to
> be not casted to other type?
>
> The latest issue with bags:
>
> a = load 'a' as (word: chararray, length: long, phrases: bag{t:
> tuple(id: chararray)});
> b = order a by word;
> store b into 'b';
>
> It gives lots of errors like:
>
> 2009-01-10 20:01:49,507 [main] ERROR
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher
> - Error message from task (map)
> task_200901101933_0007_m_000077java.lang.RuntimeException: Unexpected
> data type 116 found in stream.
> at org.apache.pig.data.DataReaderWriter.readDatum(DataReaderWriter.java:117)
> at org.apache.pig.builtin.BinStorage.getNext(BinStorage.java:90)
> at org.apache.pig.impl.builtin.RandomSampleLoader.getNext(RandomSampleLoader.java:44)
> at org.apache.pig.backend.executionengine.PigSlice.next(PigSlice.java:101)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SliceWrapper$1.next(SliceWrapper.java:157)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SliceWrapper$1.next(SliceWrapper.java:133)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.Child.main(Child.java:155)
>
> The number in "Unexpected data type 116 found in stream" message varies.
>
> 2009-01-10 20:01:49,507 [main] ERROR
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher
> - Error message from task (map)
> task_200901101933_0007_m_000078java.lang.OutOfMemoryError: Java heap
> space
> at org.apache.pig.data.DataReaderWriter.readDatum(DataReaderWriter.java:105)
> at org.apache.pig.builtin.BinStorage.getNext(BinStorage.java:90)
> at org.apache.pig.impl.builtin.RandomSampleLoader.getNext(RandomSampleLoader.java:44)
> at org.apache.pig.backend.executionengine.PigSlice.next(PigSlice.java:101)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SliceWrapper$1.next(SliceWrapper.java:157)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SliceWrapper$1.next(SliceWrapper.java:133)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.Child.main(Child.java:155)
>
> The data files are not very large and mapred.child.java.opts options
> is -Xmx2048m.
>
> If column 'phrases' is filtered out before ordering, everything is ok.
>
> What is wrong with my bags usage?
>
> Thanks.
>