You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ralph Romanos <ma...@hotmail.com> on 2012/10/08 17:06:31 UTC

Problem while streaming SSTables with BulkOutputFormat

Hello,
I am using BulkOutputFormat to load data from a .csv file into Cassandra. I am using Cassandra 1.1.3 and Hadoop 0.20.2.I have 7 hadoop nodes: 1 namenode/jobtracker and 6 datanodes/tasktrackers. Cassandra is installed on 4 of these 6 datanodes/tasktrackers.The issue happens when I have more than 1 reducer, SSTables are generated in each node, however, I get the following error in the tasktracker's logs when they are streamed into the Cassandra cluster:
Exception in thread "Streaming to /172.16.110.79:1" java.lang.RuntimeException: java.io.EOFException
	at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
	at java.io.DataInputStream.readInt(Unknown Source)
	at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194)
	at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:181)
	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:94)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
	... 3 more
Exception in thread "Streaming to /172.16.110.92:1" java.lang.RuntimeException: java.io.EOFException
	at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
	at java.io.DataInputStream.readInt(Unknown Source)
	at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194)
	at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:181)
	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:94)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
	... 3 more ...
This is what I get in the logs of one of my Cassandra nodes:ERROR 16:47:34,904 Sending retry message failed, closing session.
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcher.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(Unknown Source)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
        at sun.nio.ch.IOUtil.write(Unknown Source)
        at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
        at java.nio.channels.Channels.writeFullyImpl(Unknown Source)
        at java.nio.channels.Channels.writeFully(Unknown Source)
        at java.nio.channels.Channels.access$000(Unknown Source)
        at java.nio.channels.Channels$1.write(Unknown Source)
        at java.io.OutputStream.write(Unknown Source)
        at java.nio.channels.Channels$1.write(Unknown Source)
        at java.io.DataOutputStream.writeInt(Unknown Source)
        at org.apache.cassandra.net.OutboundTcpConnection.write(OutboundTcpConnection.java:196)
        at org.apache.cassandra.streaming.StreamInSession.sendMessage(StreamInSession.java:171)
        at org.apache.cassandra.streaming.StreamInSession.retry(StreamInSession.java:160)
        at org.apache.cassandra.streaming.IncomingStreamReader.retry(IncomingStreamReader.java:168)
        at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:98)
        at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:182)
        at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)

Does anyone know what caused these errors?
Thank you for your help.Regards,Ralph

RE: Problem while streaming SSTables with BulkOutputFormat

Posted by Ralph Romanos <ma...@hotmail.com>.
Hello again,
I noticed that this issue happens whenever a reduce task is done (so the SSTable is generated) while an SSTable already generated is being streamed to the cluster. I think that the error is therefore caused because cassandra cannot queue SSTables that are streamed to the cluster.Does that make sense?
Cheers,Ralph

From: matganine@hotmail.com
To: user@cassandra.apache.org
Subject: RE: Problem while streaming SSTables with BulkOutputFormat
Date: Tue, 9 Oct 2012 22:29:41 +0000





Aaron,Thank you for your answer, I tried to move to Cassandra 1.1.5, but the error still occurs.When I set a single task or less per hadoop node, the error does not happen.However, when I have more than one task on any of the nodes (Hadoop only node or Hadoop+Cassandra node),the error happens. When it happens, the task fails and is sent after a while to another node and completed. Ultimately, I getall my tasks done but it takes much more time.
Is it possible that streaming multiple SSTables generated from two different tasks done by the same node to the Cassandra cluster is thecause of this issue?
CheersRalph
> Subject: Re: Problem while streaming SSTables with BulkOutputFormat
> From: aaron@thelastpickle.com
> Date: Wed, 10 Oct 2012 10:05:13 +1300
> To: user@cassandra.apache.org
> 
> Something, somewhere, at some point is breaking the connection. Sorry I cannot be of more help :)
> 
> Something caused the streaming to fail, which started a retry, which failed because the pipe was broken. 
> 
> Are there any earlier errors in the logs ? 
> Did this happen on one of the nodes that has both a task tacker and cassandra ?
> 
> Cheers
> 
> 
> On 9/10/2012, at 4:06 AM, Ralph Romanos <ma...@hotmail.com> wrote:
> 
> > Hello,
> > 
> > I am using BulkOutputFormat to load data from a .csv file into Cassandra. I am using Cassandra 1.1.3 and Hadoop 0.20.2.
> > I have 7 hadoop nodes: 1 namenode/jobtracker and 6 datanodes/tasktrackers. Cassandra is installed on 4 of these 6 datanodes/tasktrackers.
> > The issue happens when I have more than 1 reducer, SSTables are generated in each node, however, I get the following error in the tasktracker's logs when they 
> > are streamed into the Cassandra cluster:
> > 
> > Exception in thread "Streaming to /172.16.110.79:1" java.lang.RuntimeException: java.io.EOFException
> > 	at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
> > 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> > 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> > 	at java.lang.Thread.run(Unknown Source)
> > Caused by: java.io.EOFException
> > 	at java.io.DataInputStream.readInt(Unknown Source)
> > 	at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194)
> > 	at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:181)
> > 	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:94)
> > 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> > 	... 3 more
> > Exception in thread "Streaming to /172.16.110.92:1" java.lang.RuntimeException: java.io.EOFException
> > 	at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
> > 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> > 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> > 	at java.lang.Thread.run(Unknown Source)
> > Caused by: java.io.EOFException
> > 	at java.io.DataInputStream.readInt(Unknown Source)
> > 	at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194)
> > 	at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:181)
> > 	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:94)
> > 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> > 	... 3 more
> > 
> >  ...
> > 
> > This is what I get in the logs of one of my Cassandra nodes:
> > ERROR 16:47:34,904 Sending retry message failed, closing session.
> > java.io.IOException: Broken pipe
> >         at sun.nio.ch.FileDispatcher.write0(Native Method)
> >         at sun.nio.ch.SocketDispatcher.write(Unknown Source)
> >         at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
> >         at sun.nio.ch.IOUtil.write(Unknown Source)
> >         at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
> >         at java.nio.channels.Channels.writeFullyImpl(Unknown Source)
> >         at java.nio.channels.Channels.writeFully(Unknown Source)
> >         at java.nio.channels.Channels.access$000(Unknown Source)
> >         at java.nio.channels.Channels$1.write(Unknown Source)
> >         at java.io.OutputStream.write(Unknown Source)
> >         at java.nio.channels.Channels$1.write(Unknown Source)
> >         at java.io.DataOutputStream.writeInt(Unknown Source)
> >         at org.apache.cassandra.net.OutboundTcpConnection.write(OutboundTcpConnection.java:196)
> >         at org.apache.cassandra.streaming.StreamInSession.sendMessage(StreamInSession.java:171)
> >         at org.apache.cassandra.streaming.StreamInSession.retry(StreamInSession.java:160)
> >         at org.apache.cassandra.streaming.IncomingStreamReader.retry(IncomingStreamReader.java:168)
> >         at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:98)
> >         at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:182)
> >         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
> > 
> > 
> > Does anyone know what caused these errors?
> > 
> > Thank you for your help.
> > Regards,
> > Ralph
> 
> 
> 
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>

RE: Problem while streaming SSTables with BulkOutputFormat

Posted by Ralph Romanos <ma...@hotmail.com>.
Aaron,Thank you for your answer, I tried to move to Cassandra 1.1.5, but the error still occurs.When I set a single task or less per hadoop node, the error does not happen.However, when I have more than one task on any of the nodes (Hadoop only node or Hadoop+Cassandra node),the error happens. When it happens, the task fails and is sent after a while to another node and completed. Ultimately, I getall my tasks done but it takes much more time.
Is it possible that streaming multiple SSTables generated from two different tasks done by the same node to the Cassandra cluster is thecause of this issue?
CheersRalph
> Subject: Re: Problem while streaming SSTables with BulkOutputFormat
> From: aaron@thelastpickle.com
> Date: Wed, 10 Oct 2012 10:05:13 +1300
> To: user@cassandra.apache.org
> 
> Something, somewhere, at some point is breaking the connection. Sorry I cannot be of more help :)
> 
> Something caused the streaming to fail, which started a retry, which failed because the pipe was broken. 
> 
> Are there any earlier errors in the logs ? 
> Did this happen on one of the nodes that has both a task tacker and cassandra ?
> 
> Cheers
> 
> 
> On 9/10/2012, at 4:06 AM, Ralph Romanos <ma...@hotmail.com> wrote:
> 
> > Hello,
> > 
> > I am using BulkOutputFormat to load data from a .csv file into Cassandra. I am using Cassandra 1.1.3 and Hadoop 0.20.2.
> > I have 7 hadoop nodes: 1 namenode/jobtracker and 6 datanodes/tasktrackers. Cassandra is installed on 4 of these 6 datanodes/tasktrackers.
> > The issue happens when I have more than 1 reducer, SSTables are generated in each node, however, I get the following error in the tasktracker's logs when they 
> > are streamed into the Cassandra cluster:
> > 
> > Exception in thread "Streaming to /172.16.110.79:1" java.lang.RuntimeException: java.io.EOFException
> > 	at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
> > 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> > 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> > 	at java.lang.Thread.run(Unknown Source)
> > Caused by: java.io.EOFException
> > 	at java.io.DataInputStream.readInt(Unknown Source)
> > 	at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194)
> > 	at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:181)
> > 	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:94)
> > 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> > 	... 3 more
> > Exception in thread "Streaming to /172.16.110.92:1" java.lang.RuntimeException: java.io.EOFException
> > 	at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
> > 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> > 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> > 	at java.lang.Thread.run(Unknown Source)
> > Caused by: java.io.EOFException
> > 	at java.io.DataInputStream.readInt(Unknown Source)
> > 	at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194)
> > 	at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:181)
> > 	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:94)
> > 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> > 	... 3 more
> > 
> >  ...
> > 
> > This is what I get in the logs of one of my Cassandra nodes:
> > ERROR 16:47:34,904 Sending retry message failed, closing session.
> > java.io.IOException: Broken pipe
> >         at sun.nio.ch.FileDispatcher.write0(Native Method)
> >         at sun.nio.ch.SocketDispatcher.write(Unknown Source)
> >         at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
> >         at sun.nio.ch.IOUtil.write(Unknown Source)
> >         at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
> >         at java.nio.channels.Channels.writeFullyImpl(Unknown Source)
> >         at java.nio.channels.Channels.writeFully(Unknown Source)
> >         at java.nio.channels.Channels.access$000(Unknown Source)
> >         at java.nio.channels.Channels$1.write(Unknown Source)
> >         at java.io.OutputStream.write(Unknown Source)
> >         at java.nio.channels.Channels$1.write(Unknown Source)
> >         at java.io.DataOutputStream.writeInt(Unknown Source)
> >         at org.apache.cassandra.net.OutboundTcpConnection.write(OutboundTcpConnection.java:196)
> >         at org.apache.cassandra.streaming.StreamInSession.sendMessage(StreamInSession.java:171)
> >         at org.apache.cassandra.streaming.StreamInSession.retry(StreamInSession.java:160)
> >         at org.apache.cassandra.streaming.IncomingStreamReader.retry(IncomingStreamReader.java:168)
> >         at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:98)
> >         at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:182)
> >         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
> > 
> > 
> > Does anyone know what caused these errors?
> > 
> > Thank you for your help.
> > Regards,
> > Ralph
> 
> 
> 
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>

Re: Problem while streaming SSTables with BulkOutputFormat

Posted by aaron morton <aa...@thelastpickle.com>.
Something, somewhere, at some point is breaking the connection. Sorry I cannot be of more help :)

Something caused the streaming to fail, which started a retry, which failed because the pipe was broken. 

Are there any earlier errors in the logs ? 
Did this happen on one of the nodes that has both a task tacker and cassandra ?

Cheers


On 9/10/2012, at 4:06 AM, Ralph Romanos <ma...@hotmail.com> wrote:

> Hello,
> 
> I am using BulkOutputFormat to load data from a .csv file into Cassandra. I am using Cassandra 1.1.3 and Hadoop 0.20.2.
> I have 7 hadoop nodes: 1 namenode/jobtracker and 6 datanodes/tasktrackers. Cassandra is installed on 4 of these 6 datanodes/tasktrackers.
> The issue happens when I have more than 1 reducer, SSTables are generated in each node, however, I get the following error in the tasktracker's logs when they 
> are streamed into the Cassandra cluster:
> 
> Exception in thread "Streaming to /172.16.110.79:1" java.lang.RuntimeException: java.io.EOFException
> 	at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> 	at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> 	at java.io.DataInputStream.readInt(Unknown Source)
> 	at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194)
> 	at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:181)
> 	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:94)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> 	... 3 more
> Exception in thread "Streaming to /172.16.110.92:1" java.lang.RuntimeException: java.io.EOFException
> 	at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> 	at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> 	at java.io.DataInputStream.readInt(Unknown Source)
> 	at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194)
> 	at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:181)
> 	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:94)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> 	... 3 more
> 
>  ...
> 
> This is what I get in the logs of one of my Cassandra nodes:
> ERROR 16:47:34,904 Sending retry message failed, closing session.
> java.io.IOException: Broken pipe
>         at sun.nio.ch.FileDispatcher.write0(Native Method)
>         at sun.nio.ch.SocketDispatcher.write(Unknown Source)
>         at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
>         at sun.nio.ch.IOUtil.write(Unknown Source)
>         at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
>         at java.nio.channels.Channels.writeFullyImpl(Unknown Source)
>         at java.nio.channels.Channels.writeFully(Unknown Source)
>         at java.nio.channels.Channels.access$000(Unknown Source)
>         at java.nio.channels.Channels$1.write(Unknown Source)
>         at java.io.OutputStream.write(Unknown Source)
>         at java.nio.channels.Channels$1.write(Unknown Source)
>         at java.io.DataOutputStream.writeInt(Unknown Source)
>         at org.apache.cassandra.net.OutboundTcpConnection.write(OutboundTcpConnection.java:196)
>         at org.apache.cassandra.streaming.StreamInSession.sendMessage(StreamInSession.java:171)
>         at org.apache.cassandra.streaming.StreamInSession.retry(StreamInSession.java:160)
>         at org.apache.cassandra.streaming.IncomingStreamReader.retry(IncomingStreamReader.java:168)
>         at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:98)
>         at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:182)
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
> 
> 
> Does anyone know what caused these errors?
> 
> Thank you for your help.
> Regards,
> Ralph



-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com