You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by "Johannes.Lichtenberger" <Jo...@uni-konstanz.de> on 2010/10/07 05:02:36 UTC
ClassCastException
Hello,
I'm getting a ClassCastException, when running my application:
10/10/07 04:31:10 WARN mapred.LocalJobRunner: job_local_0001
java.lang.ClassCastException: class java.util.Date
at java.lang.Class.asSubclass(Class.java:3018)
at
org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:617)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:791)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:524)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
My key is a date (java.util.Date.class).
final Job job = new Job(getConf());
...
job.setOutputKeyClass(java.util.Date.class);
job.setOutputValueClass(java.util.List.class);
What's wrong?
regards,
Johannes
Re: ClassCastException
Posted by "Johannes.Lichtenberger" <Jo...@uni-konstanz.de>.
On 10/08/2010 02:52 AM, Ted Yu wrote:
> Have you checked
> http://download.oracle.com/javase/6/docs/api/javax/xml/stream/XMLEventReader.html?
Hm, I'm working really often with StAX and I'm using the event reader in
my custom input format to create the records and produce a List of
XMLEvents or more precisely now ListWritable which has a membervariable
List<XMLEventWritable> and extends ArrayWritable:
/**
* Constructor.
*
* @param paramList
* List of {@link XMLEvent}s.
*/
public ListWritable(final List<XMLEventWritable> paramList) {
super(XMLEventWritable.class, paramList.toArray(new
XMLEventWritable[paramList.size()]));
mList = paramList;
}
...
and in my XMLEventWritable:
@Override
public void write(final DataOutput paramOut) throws IOException {
try {
mEvent.writeAsEncodedUnicode(mWriter);
} catch (final XMLStreamException e) {
// Implement logging.
}
mWriter.flush();
paramOut.writeUTF(mWriter.toString());
}
But my problem is now readFields(final DataInput paramIn). I assume I
have to create an XMLEventFactory, but I'm not sure how to determine the
type of the event and so on (normally event.getEventType()). So I would
have to get the whole input:
final String line = paramIn.readUTF();
and parse it if it's a start tag, end tag, comment, character... and
then create the appropriate event?
Baaah, horrible ;-)
Re: ClassCastException
Posted by Ted Yu <yu...@gmail.com>.
Have you checked
http://download.oracle.com/javase/6/docs/api/javax/xml/stream/XMLEventReader.html?
On Thu, Oct 7, 2010 at 5:49 PM, Johannes.Lichtenberger <
Johannes.Lichtenberger@uni-konstanz.de> wrote:
> On 10/08/2010 02:38 AM, Johannes.Lichtenberger wrote:
> > On 10/08/2010 01:29 AM, Ted Yu wrote:
> >>
> http://download.oracle.com/javase/6/docs/api/javax/xml/stream/XMLEventWriter.html
> >>
> >> You can use an XMLOutputFactory to create an XMLEventWriter, and then
> use an
> >> XMLEventFactory to create events that can then be written to the
> >> XMLEventWriter.
> >
> > I just rethought about the List with Writable XMLEvents. I think it
> > would probably be better, otherwise I have to create an XMLEventReader
> > in the RecordReader, Mapper and Reducer, and everytime convert the
> > events back to Text, whereas a List of XMLEvents would be sufficient.
> > I'm currently not sure.
>
> Hm, maybe I can just call mEvent.writeAsEncodedUnicode(writer), than
> output the writer but then I'm not entirely sure how to do the reverse
> thing, reading the input (how to implement readFields(...)).
>
> regards,
> Johannes
>
Re: ClassCastException
Posted by "Johannes.Lichtenberger" <Jo...@uni-konstanz.de>.
On 10/08/2010 02:38 AM, Johannes.Lichtenberger wrote:
> On 10/08/2010 01:29 AM, Ted Yu wrote:
>> http://download.oracle.com/javase/6/docs/api/javax/xml/stream/XMLEventWriter.html
>>
>> You can use an XMLOutputFactory to create an XMLEventWriter, and then use an
>> XMLEventFactory to create events that can then be written to the
>> XMLEventWriter.
>
> I just rethought about the List with Writable XMLEvents. I think it
> would probably be better, otherwise I have to create an XMLEventReader
> in the RecordReader, Mapper and Reducer, and everytime convert the
> events back to Text, whereas a List of XMLEvents would be sufficient.
> I'm currently not sure.
Hm, maybe I can just call mEvent.writeAsEncodedUnicode(writer), than
output the writer but then I'm not entirely sure how to do the reverse
thing, reading the input (how to implement readFields(...)).
regards,
Johannes
Re: ClassCastException
Posted by "Johannes.Lichtenberger" <Jo...@uni-konstanz.de>.
On 10/08/2010 01:29 AM, Ted Yu wrote:
> http://download.oracle.com/javase/6/docs/api/javax/xml/stream/XMLEventWriter.html
>
> You can use an XMLOutputFactory to create an XMLEventWriter, and then use an
> XMLEventFactory to create events that can then be written to the
> XMLEventWriter.
I just rethought about the List with Writable XMLEvents. I think it
would probably be better, otherwise I have to create an XMLEventReader
in the RecordReader, Mapper and Reducer, and everytime convert the
events back to Text, whereas a List of XMLEvents would be sufficient.
I'm currently not sure.
regards,
Johannes
Re: ClassCastException
Posted by Ted Yu <yu...@gmail.com>.
http://download.oracle.com/javase/6/docs/api/javax/xml/stream/XMLEventWriter.html
You can use an XMLOutputFactory to create an XMLEventWriter, and then use an
XMLEventFactory to create events that can then be written to the
XMLEventWriter.
On Thu, Oct 7, 2010 at 4:05 PM, Johannes.Lichtenberger <
Johannes.Lichtenberger@uni-konstanz.de> wrote:
> On 10/08/2010 12:01 AM, Ted Yu wrote:
> > http://www.ibm.com/developerworks/xml/library/x-stax2.html
>
> I think the problem would be how to serialize XMLEvents or more
> precisely I don't know if there's an existing StAX XMLEvent-to-String
> class/method.
>
> regards,
> Johannes
>
>
Re: ClassCastException
Posted by "Johannes.Lichtenberger" <Jo...@uni-konstanz.de>.
On 10/08/2010 12:01 AM, Ted Yu wrote:
> http://www.ibm.com/developerworks/xml/library/x-stax2.html
I think the problem would be how to serialize XMLEvents or more
precisely I don't know if there's an existing StAX XMLEvent-to-String
class/method.
regards,
Johannes
Re: ClassCastException
Posted by "Johannes.Lichtenberger" <Jo...@uni-konstanz.de>.
On 10/08/2010 12:01 AM, Ted Yu wrote:
> http://www.ibm.com/developerworks/xml/library/x-stax2.html
Yes, my approach is to parse a very big XML file (wikipedia revisions)
with StAX in my RecordReader implementation. The key is a timestamp and
the values are List<XMLEventWritable>s, because I don't want to have to
setup a new StAX-Parser in every Map, but ok, I assume the cost of
setting up StAX-Parsers is negligible, so I can write Text-values
instead of List<XMLEventWritable>-values.
regards,
Johannes
Re: ClassCastException
Posted by Ted Yu <yu...@gmail.com>.
http://www.ibm.com/developerworks/xml/library/x-stax2.html
On Thu, Oct 7, 2010 at 2:54 PM, Johannes.Lichtenberger <
Johannes.Lichtenberger@uni-konstanz.de> wrote:
> On 10/07/2010 05:41 PM, Ted Yu wrote:
> > Since mFormatter.format() returns a String, you don't need to introduce
> > newline.
> > You can call paramOut.writeUTF() to save the String and call
> > paramIn.readUTF() to read it back.
>
> Ok, that's what I did right after replying. My value is a List<XMLEvent>
> or more precisely now it's ListWritable which extends ArrayWritable:
>
> /** {@link List} of {@link XMLEventWritable}s. */
> private List<XMLEventWritable> mList;
>
> /**
> * Constructor.
> *
> * @param paramList
> * List of {@link XMLEvent}s.
> */
> public ListWritable(final List<XMLEventWritable> paramList) {
> super(XMLEventWritable.class, paramList.toArray(new
> XMLEventWritable[paramList.size()]));
> mList = paramList;
> }
>
> @Override
> public void readFields(final DataInput paramIn) throws IOException {
> readFields(paramIn);
> mList = Arrays.asList((XMLEventWritable[])get());
> }
>
> I'm currently writing the XMLEventWritable class, which will be a
> slightly bigger thing (serialization depends on the EventType). I
> haven't coded any serialization/deserialization of Objects before, so
> I've for example something like
>
> @Override
> public void write(final DataOutput paramOut) throws IOException {
> paramOut.write(mEvent.getEventType());
> paramOut.writeChar(';');
>
> switch (mEvent.getEventType()) {
> case XMLStreamConstants.START_DOCUMENT:
> break;
> case XMLStreamConstants.START_ELEMENT:
> try {
> // Write start tag.
> final StartElement element = mEvent.asStartElement();
> final QName name = element.getName();
> final String prefix = name.getPrefix();
> if (prefix != null && prefix.length() > 0) {
> paramOut.writeUTF(prefix);
> paramOut.writeChar(':');
> }
> paramOut.writeUTF(name.getLocalPart());
> ...
>
> I wonder what to write in the read(...) method. I assume I have to read
> in the whole String via paramIn.readUTF() and then parse it and build a
> new XMLEvent. Ah well, I assume no serialization/deserialization class
> of any class which implements XMLEvent
> (
> http://download-llnw.oracle.com/javase/6/docs/api/javax/xml/stream/events/XMLEvent.html
> )
> does exist!?
>
> regards,
> Johannes
>
Re: ClassCastException
Posted by "Johannes.Lichtenberger" <Jo...@uni-konstanz.de>.
On 10/07/2010 05:41 PM, Ted Yu wrote:
> Since mFormatter.format() returns a String, you don't need to introduce
> newline.
> You can call paramOut.writeUTF() to save the String and call
> paramIn.readUTF() to read it back.
Ok, that's what I did right after replying. My value is a List<XMLEvent>
or more precisely now it's ListWritable which extends ArrayWritable:
/** {@link List} of {@link XMLEventWritable}s. */
private List<XMLEventWritable> mList;
/**
* Constructor.
*
* @param paramList
* List of {@link XMLEvent}s.
*/
public ListWritable(final List<XMLEventWritable> paramList) {
super(XMLEventWritable.class, paramList.toArray(new
XMLEventWritable[paramList.size()]));
mList = paramList;
}
@Override
public void readFields(final DataInput paramIn) throws IOException {
readFields(paramIn);
mList = Arrays.asList((XMLEventWritable[])get());
}
I'm currently writing the XMLEventWritable class, which will be a
slightly bigger thing (serialization depends on the EventType). I
haven't coded any serialization/deserialization of Objects before, so
I've for example something like
@Override
public void write(final DataOutput paramOut) throws IOException {
paramOut.write(mEvent.getEventType());
paramOut.writeChar(';');
switch (mEvent.getEventType()) {
case XMLStreamConstants.START_DOCUMENT:
break;
case XMLStreamConstants.START_ELEMENT:
try {
// Write start tag.
final StartElement element = mEvent.asStartElement();
final QName name = element.getName();
final String prefix = name.getPrefix();
if (prefix != null && prefix.length() > 0) {
paramOut.writeUTF(prefix);
paramOut.writeChar(':');
}
paramOut.writeUTF(name.getLocalPart());
...
I wonder what to write in the read(...) method. I assume I have to read
in the whole String via paramIn.readUTF() and then parse it and build a
new XMLEvent. Ah well, I assume no serialization/deserialization class
of any class which implements XMLEvent
(http://download-llnw.oracle.com/javase/6/docs/api/javax/xml/stream/events/XMLEvent.html)
does exist!?
regards,
Johannes
Re: ClassCastException
Posted by Ted Yu <yu...@gmail.com>.
Since mFormatter.format() returns a String, you don't need to introduce
newline.
You can call paramOut.writeUTF() to save the String and call
paramIn.readUTF() to read it back.
The value class doesn't need to implement Comparable.
On Thu, Oct 7, 2010 at 6:58 AM, Johannes.Lichtenberger <
Johannes.Lichtenberger@uni-konstanz.de> wrote:
> On 10/07/2010 06:46 AM, Anthony Urso wrote:
> > Hadoop is attempting to cast a Date object to WritableComparable, which
> > Date does not implement, and is causing that exception.
> >
> > Your keys must implement WritableComparable and your values must
> > implement Comparable.
>
> The values have to implement only Comparable or Writable?
>
> So I'm currently not sure if this is right:
>
> /** {@link DateFormat}. */
> private final DateFormat mFormatter = new
> SimpleDateFormat("yyyy.MM.ddTHH.mm.ssZ", Locale.ENGLISH);
>
> /** {@link Date} used as the key. */
> private transient Date mTimestamp;
>
> /** Set timestamp. */
> public void setTimestamp(final Date paramTimestamp) {
> mTimestamp = paramTimestamp;
> }
>
> @Override
> public void readFields(final DataInput paramIn) throws IOException {
> try {
> mTimestamp = mFormatter.parse(paramIn.readLine());
> } catch (final ParseException e) {
> LOGWRAPPER.error(e.getMessage(), e);
> }
> }
>
> @Override
> public void write(final DataOutput paramOut) throws IOException {
> paramOut.writeBytes(mFormatter.format(mTimestamp));
> }
>
> Then I'm just storing a Date (timestamp) with setTimestamp(...), but I'm
> not sure if readFields(...) is implemented correctly (because of
> paramIn.readLine()). I assume I have to append a newline in write(...).
>
> @Override
> public void write(final DataOutput paramOut) throws IOException {
>
> paramOut.writeBytes(mFormatter.format(mTimestamp)+System.getProperty("line.separator"));
> }
>
> BTW: Why does a value have to implement Comparable? It's not used for
> sorting and grouping!?
>
> regards,
> Johannes
>
Re: ClassCastException
Posted by Anthony Urso <an...@cs.ucla.edu>.
Oops, I meant Writable.
On Thu, Oct 7, 2010 at 6:58 AM, Johannes.Lichtenberger
<Jo...@uni-konstanz.de> wrote:
> On 10/07/2010 06:46 AM, Anthony Urso wrote:
>> Hadoop is attempting to cast a Date object to WritableComparable, which
>> Date does not implement, and is causing that exception.
>>
>> Your keys must implement WritableComparable and your values must
>> implement Comparable.
>
> The values have to implement only Comparable or Writable?
>
> So I'm currently not sure if this is right:
>
> /** {@link DateFormat}. */
> private final DateFormat mFormatter = new
> SimpleDateFormat("yyyy.MM.ddTHH.mm.ssZ", Locale.ENGLISH);
>
> /** {@link Date} used as the key. */
> private transient Date mTimestamp;
>
> /** Set timestamp. */
> public void setTimestamp(final Date paramTimestamp) {
> mTimestamp = paramTimestamp;
> }
>
> @Override
> public void readFields(final DataInput paramIn) throws IOException {
> try {
> mTimestamp = mFormatter.parse(paramIn.readLine());
> } catch (final ParseException e) {
> LOGWRAPPER.error(e.getMessage(), e);
> }
> }
>
> @Override
> public void write(final DataOutput paramOut) throws IOException {
> paramOut.writeBytes(mFormatter.format(mTimestamp));
> }
>
> Then I'm just storing a Date (timestamp) with setTimestamp(...), but I'm
> not sure if readFields(...) is implemented correctly (because of
> paramIn.readLine()). I assume I have to append a newline in write(...).
>
> @Override
> public void write(final DataOutput paramOut) throws IOException {
> paramOut.writeBytes(mFormatter.format(mTimestamp)+System.getProperty("line.separator"));
> }
>
> BTW: Why does a value have to implement Comparable? It's not used for
> sorting and grouping!?
>
> regards,
> Johannes
>
Re: ClassCastException
Posted by "Johannes.Lichtenberger" <Jo...@uni-konstanz.de>.
On 10/07/2010 06:46 AM, Anthony Urso wrote:
> Hadoop is attempting to cast a Date object to WritableComparable, which
> Date does not implement, and is causing that exception.
>
> Your keys must implement WritableComparable and your values must
> implement Comparable.
The values have to implement only Comparable or Writable?
So I'm currently not sure if this is right:
/** {@link DateFormat}. */
private final DateFormat mFormatter = new
SimpleDateFormat("yyyy.MM.ddTHH.mm.ssZ", Locale.ENGLISH);
/** {@link Date} used as the key. */
private transient Date mTimestamp;
/** Set timestamp. */
public void setTimestamp(final Date paramTimestamp) {
mTimestamp = paramTimestamp;
}
@Override
public void readFields(final DataInput paramIn) throws IOException {
try {
mTimestamp = mFormatter.parse(paramIn.readLine());
} catch (final ParseException e) {
LOGWRAPPER.error(e.getMessage(), e);
}
}
@Override
public void write(final DataOutput paramOut) throws IOException {
paramOut.writeBytes(mFormatter.format(mTimestamp));
}
Then I'm just storing a Date (timestamp) with setTimestamp(...), but I'm
not sure if readFields(...) is implemented correctly (because of
paramIn.readLine()). I assume I have to append a newline in write(...).
@Override
public void write(final DataOutput paramOut) throws IOException {
paramOut.writeBytes(mFormatter.format(mTimestamp)+System.getProperty("line.separator"));
}
BTW: Why does a value have to implement Comparable? It's not used for
sorting and grouping!?
regards,
Johannes
Re: ClassCastException
Posted by Anthony Urso <an...@cs.ucla.edu>.
Hadoop is attempting to cast a Date object to WritableComparable, which
Date does not implement, and is causing that exception.
Your keys must implement WritableComparable and your values must
implement Comparable.
On Wed, Oct 6, 2010 at 8:02 PM, Johannes.Lichtenberger
<Jo...@uni-konstanz.de> wrote:
> Hello,
>
> I'm getting a ClassCastException, when running my application:
>
> 10/10/07 04:31:10 WARN mapred.LocalJobRunner: job_local_0001
> java.lang.ClassCastException: class java.util.Date
> at java.lang.Class.asSubclass(Class.java:3018)
> at
> org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:617)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:791)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:524)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
>
> My key is a date (java.util.Date.class).
>
> final Job job = new Job(getConf());
> ...
> job.setOutputKeyClass(java.util.Date.class);
> job.setOutputValueClass(java.util.List.class);
>
> What's wrong?
>
> regards,
> Johannes
>