You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Yuki Morishita (JIRA)" <ji...@apache.org> on 2015/02/12 18:02:11 UTC
[jira] [Comment Edited] (CASSANDRA-8774) BulkOutputFormat never
completes if streaming have errors
[ https://issues.apache.org/jira/browse/CASSANDRA-8774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318549#comment-14318549 ]
Yuki Morishita edited comment on CASSANDRA-8774 at 2/12/15 5:01 PM:
--------------------------------------------------------------------
What I understand from that {{try ... catch}} clause is that it is using TimeoutException to report progress every 1000 ms.
It looks to me that doing correct error handling when ExecutorException is thrown is the right way to do here, instead of handling StreamingException individually.
WDYT [~alexliu68]?
was (Author: yukim):
What I understand from that {{{try ... catch}}} clause is that it is using TimeoutException to report progress every 1000 ms.
It looks to me that doing correct error handling when ExecutorException is thrown is the right way to do here, instead of handling StreamingException individually.
WDYT [~alexliu68]?
> BulkOutputFormat never completes if streaming have errors
> ---------------------------------------------------------
>
> Key: CASSANDRA-8774
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8774
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Reporter: Erik Forsberg
> Fix For: 2.0.13
>
> Attachments: 0001-CASSANDRA-8774-Handle-StreamException-when-bulkloadi.patch
>
>
> With BulkoutputFormat in Cassandra 1.2.18, if any streaming errors occured, the hadoop task would fail. This doesn't seem to happen with 2.0.12.
> I have a hadoop map task that use BulkoutputFormat. If one of the cassandra nodes I'm writing to is down, I'm getting the following syslog output from the map task:
> {noformat}
> 2015-02-10 10:54:15,162 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
> 2015-02-10 10:54:15,601 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
> 2015-02-10 10:54:15,901 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
> 2015-02-10 10:54:15,907 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4984451e
> 2015-02-10 10:54:16,110 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hdpmt01.osp-hadoop.osa:9000/user/jenkins/syst/5ef13_osp/tvstore/sumcombinations/hourly/2015021002/per_period-5ba2faa4b11111e4aa21fa163e82bc46-sumcombinations/0/data/part-00047:0+462
> 2015-02-10 10:54:16,739 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
> 2015-02-10 10:54:16,740 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2015-02-10 10:54:16,927 ERROR org.apache.cassandra.cql3.QueryProcessor: Unable to initialize MemoryMeter (jamm not specified as javaagent). This means Cassandra will be unable to measure object sizes accurately and may consequently OOM.
> 2015-02-10 10:54:17,780 INFO org.apache.cassandra.utils.CLibrary: JNA not found. Native methods will be disabled.
> 2015-02-10 10:54:19,446 INFO org.apache.cassandra.io.sstable.SSTableReader: Opening /opera/log1/hadoop/mapred/local/taskTracker/jenkins/jobcache/job_201502041226_13903/attempt_201502041226_13903_m_000000_0/work/tmp/syst5ef13osp/Data_hourly/syst5ef13osp-Data_hourly-jb-1 (1018 bytes)
> 2015-02-10 10:54:20,713 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Executing streaming plan for Bulk Load
> 2015-02-10 10:54:20,713 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:7
> 2015-02-10 10:54:20,714 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:8
> 2015-02-10 10:54:20,715 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to /ipv6:prefix:1:441:0:0:0:7
> 2015-02-10 10:54:20,730 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:4
> 2015-02-10 10:54:20,750 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:3
> 2015-02-10 10:54:20,731 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to /ipv6:prefix:1:441:0:0:0:8
> 2015-02-10 10:54:20,750 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to /ipv6:prefix:1:441:0:0:0:4
> 2015-02-10 10:54:20,770 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:6
> 2015-02-10 10:54:20,778 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:5
> 2015-02-10 10:54:20,786 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to /ipv6:prefix:1:441:0:0:0:3
> 2015-02-10 10:54:20,790 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to /ipv6:prefix:1:441:0:0:0:6
> 2015-02-10 10:54:20,867 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to /ipv6:prefix:1:441:0:0:0:5
> 2015-02-10 10:54:20,897 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Prepare completed. Receiving 0 files(0 bytes), sending 1 files(152 bytes)
> 2015-02-10 10:54:20,897 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Prepare completed. Receiving 0 files(0 bytes), sending 1 files(491 bytes)
> 2015-02-10 10:54:20,897 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Prepare completed. Receiving 0 files(0 bytes), sending 1 files(375 bytes)
> 2015-02-10 10:54:20,897 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Prepare completed. Receiving 0 files(0 bytes), sending 1 files(322 bytes)
> 2015-02-10 10:54:20,898 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Prepare completed. Receiving 0 files(0 bytes), sending 1 files(169 bytes)
> 2015-02-10 10:54:20,983 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:7 is complete
> 2015-02-10 10:54:20,984 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:8 is complete
> 2015-02-10 10:54:20,985 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:3 is complete
> 2015-02-10 10:54:20,997 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:4 is complete
> 2015-02-10 10:54:21,067 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:6 is complete
> 2015-02-10 10:55:24,027 WARN org.apache.cassandra.streaming.DefaultConnectionFactory: Failed attempt 1 to connect to /ipv6:prefix:1:441:0:0:0:5. Retrying in 20000 ms. (java.net.ConnectException: Connection timed out)
> 2015-02-10 10:56:54,344 WARN org.apache.cassandra.streaming.DefaultConnectionFactory: Failed attempt 2 to connect to /ipv6:prefix:1:441:0:0:0:5. Retrying in 40000 ms. (java.net.ConnectException: Connection timed out)
> 2015-02-10 10:58:37,493 ERROR org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Streaming error occurred
> java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect0(Native Method)
> at sun.nio.ch.Net.connect(Net.java:465)
> at sun.nio.ch.Net.connect(Net.java:457)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
> at java.nio.channels.SocketChannel.open(SocketChannel.java:184)
> at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:134)
> at org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:52)
> at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:235)
> at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:77)
> at org.apache.cassandra.streaming.StreamSession$1.run(StreamSession.java:221)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2015-02-10 10:58:37,502 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:5 is complete
> 2015-02-10 10:58:37,507 WARN org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Stream failed
> {noformat}
> The hadoop task will then just sit there and do nothing forever, which indicates that it's getting progress updates from the cassandra code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)