You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Brian Jeltema <br...@digitalenvoy.net> on 2013/01/30 13:19:48 UTC
cryptic exception in Hadoop/Cassandra job
I have a Hadoop/Cassandra map/reduce job that performs a simple transformation on a table
with very roughly 1 billion columns spread across roughly 4 million rows. During reduction, I
see a relative handful of the following:
Exception in thread "Streaming to /10.4.0.3:1" java.lang.RuntimeException: java.io.EOFException
at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194)
at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:104)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
which ultimately leads to job failure. I can't tell if this is a bug in my code or in the underlying framework.
Does anyone have suggestions on how to debug this?
TIA
Brian
Re: cryptic exception in Hadoop/Cassandra job
Posted by Brian Jeltema <br...@digitalenvoy.net>.
I'm not sure this is the same problem. I'm getting these even when using a single reducer
for the entire job.
Brian
On Jan 30, 2013, at 9:26 AM, Pieter Callewaert wrote:
> I have the same issue (but with sstableloaders).
> Should be fixed in 1.2 release (https://issues.apache.org/jira/browse/CASSANDRA-4813)
>
> Kind regards,
> Pieter
>
>
> -----Original Message-----
> From: Brian Jeltema [mailto:brian.jeltema@digitalenvoy.net]
> Sent: woensdag 30 januari 2013 13:58
> To: user@cassandra.apache.org
> Subject: Re: cryptic exception in Hadoop/Cassandra job
>
> Cassandra 1.1.5, using BulkOutputFormat
>
> Brian
>
> On Jan 30, 2013, at 7:39 AM, Pieter Callewaert wrote:
>
>> Hi Brian,
>>
>> Which version of cassandra are you using? And are you using the BOF to write to Cassandra?
>>
>> Kind regards,
>> Pieter
>>
>> -----Original Message-----
>> From: Brian Jeltema [mailto:brian.jeltema@digitalenvoy.net]
>> Sent: woensdag 30 januari 2013 13:20
>> To: user@cassandra.apache.org
>> Subject: cryptic exception in Hadoop/Cassandra job
>>
>>
>> I have a Hadoop/Cassandra map/reduce job that performs a simple transformation on a table with very roughly 1 billion columns spread across roughly 4 million rows. During reduction, I see a relative handful of the following:
>>
>> Exception in thread "Streaming to /10.4.0.3:1" java.lang.RuntimeException: java.io.EOFException
>> at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.io.EOFException
>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>> at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194)
>> at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:104)
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> ... 3 more
>>
>> which ultimately leads to job failure. I can't tell if this is a bug in my code or in the underlying framework.
>> Does anyone have suggestions on how to debug this?
>>
>> TIA
>>
>> Brian
>>
>>
>
>
>
>
RE: cryptic exception in Hadoop/Cassandra job
Posted by Pieter Callewaert <pi...@be-mobile.be>.
I have the same issue (but with sstableloaders).
Should be fixed in 1.2 release (https://issues.apache.org/jira/browse/CASSANDRA-4813)
Kind regards,
Pieter
-----Original Message-----
From: Brian Jeltema [mailto:brian.jeltema@digitalenvoy.net]
Sent: woensdag 30 januari 2013 13:58
To: user@cassandra.apache.org
Subject: Re: cryptic exception in Hadoop/Cassandra job
Cassandra 1.1.5, using BulkOutputFormat
Brian
On Jan 30, 2013, at 7:39 AM, Pieter Callewaert wrote:
> Hi Brian,
>
> Which version of cassandra are you using? And are you using the BOF to write to Cassandra?
>
> Kind regards,
> Pieter
>
> -----Original Message-----
> From: Brian Jeltema [mailto:brian.jeltema@digitalenvoy.net]
> Sent: woensdag 30 januari 2013 13:20
> To: user@cassandra.apache.org
> Subject: cryptic exception in Hadoop/Cassandra job
>
>
> I have a Hadoop/Cassandra map/reduce job that performs a simple transformation on a table with very roughly 1 billion columns spread across roughly 4 million rows. During reduction, I see a relative handful of the following:
>
> Exception in thread "Streaming to /10.4.0.3:1" java.lang.RuntimeException: java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194)
> at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:104)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
>
> which ultimately leads to job failure. I can't tell if this is a bug in my code or in the underlying framework.
> Does anyone have suggestions on how to debug this?
>
> TIA
>
> Brian
>
>
Re: cryptic exception in Hadoop/Cassandra job
Posted by Brian Jeltema <br...@digitalenvoy.net>.
Cassandra 1.1.5, using BulkOutputFormat
Brian
On Jan 30, 2013, at 7:39 AM, Pieter Callewaert wrote:
> Hi Brian,
>
> Which version of cassandra are you using? And are you using the BOF to write to Cassandra?
>
> Kind regards,
> Pieter
>
> -----Original Message-----
> From: Brian Jeltema [mailto:brian.jeltema@digitalenvoy.net]
> Sent: woensdag 30 januari 2013 13:20
> To: user@cassandra.apache.org
> Subject: cryptic exception in Hadoop/Cassandra job
>
>
> I have a Hadoop/Cassandra map/reduce job that performs a simple transformation on a table with very roughly 1 billion columns spread across roughly 4 million rows. During reduction, I see a relative handful of the following:
>
> Exception in thread "Streaming to /10.4.0.3:1" java.lang.RuntimeException: java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194)
> at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:104)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
>
> which ultimately leads to job failure. I can't tell if this is a bug in my code or in the underlying framework.
> Does anyone have suggestions on how to debug this?
>
> TIA
>
> Brian
>
>
RE: cryptic exception in Hadoop/Cassandra job
Posted by Pieter Callewaert <pi...@be-mobile.be>.
Hi Brian,
Which version of cassandra are you using? And are you using the BOF to write to Cassandra?
Kind regards,
Pieter
-----Original Message-----
From: Brian Jeltema [mailto:brian.jeltema@digitalenvoy.net]
Sent: woensdag 30 januari 2013 13:20
To: user@cassandra.apache.org
Subject: cryptic exception in Hadoop/Cassandra job
I have a Hadoop/Cassandra map/reduce job that performs a simple transformation on a table with very roughly 1 billion columns spread across roughly 4 million rows. During reduction, I see a relative handful of the following:
Exception in thread "Streaming to /10.4.0.3:1" java.lang.RuntimeException: java.io.EOFException
at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194)
at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:104)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
which ultimately leads to job failure. I can't tell if this is a bug in my code or in the underlying framework.
Does anyone have suggestions on how to debug this?
TIA
Brian