You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Yuki Morishita (JIRA)" <ji...@apache.org> on 2013/05/03 19:04:16 UTC
[jira] [Updated] (CASSANDRA-5229) after a IOException is thrown
during streaming, streaming tasks hang in netstats
[ https://issues.apache.org/jira/browse/CASSANDRA-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita updated CASSANDRA-5229:
--------------------------------------
Attachment: 5229-1.2.txt
I think the best we can do here is to catch RuntimeException and let session fail.
Patch attach for that.
> after a IOException is thrown during streaming, streaming tasks hang in netstats
> --------------------------------------------------------------------------------
>
> Key: CASSANDRA-5229
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5229
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.1
> Environment: Ubuntu 12.04
> Reporter: Michael Kjellman
> Assignee: Yuki Morishita
> Priority: Minor
> Fix For: 1.2.5
>
> Attachments: 5229-1.2.txt
>
>
> After an IOExcpetion, streaming tasks marked as "successful" in the logs are hung in netstats
> With TRACE debugging on streaming on the receiving node everything about the sstable in the log (not very much)
> {code}
> INFO [AntiEntropyStage:1] 2013-02-07 11:23:44,717 StreamOut.java (line 151) Stream context metadata [/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-5-Data.db sections=3068 progress=0/2785204713 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-25-Data.db sections=2696 progress=0/758409465 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-60-Data.db sections=3099 progress=0/238876436 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-63-Data.db sections=1166 progress=0/2125323 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-38-Data.db sections=2507 progress=0/515992757 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-26-Data.db sections=3153 progress=0/994857654 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-57-Data.db sections=3116 progress=0/129398170 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-58-Data.db sections=217 progress=0/72286 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-59-Data.db sections=3146 progress=0/3357709019 - 0%], 27 sstables.
> INFO [AntiEntropyStage:1] 2013-02-07 11:23:52,964 StreamOut.java (line 151) Stream context metadata [/data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-5-Data.db sections=2930 progress=0/2799914560 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-25-Data.db sections=2590 progress=0/761266059 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-60-Data.db sections=2956 progress=0/241362497 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-63-Data.db sections=1153 progress=0/2125323 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-38-Data.db sections=2422 progress=0/522126371 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-26-Data.db sections=3004 progress=0/998401202 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-57-Data.db sections=2974 progress=0/129722346 - 0%, /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-58-Data.db sections=220 progress=0/72286 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-59-Data.db sections=2998 progress=0/3375554099 - 0%], 27 sstables.
> {code}
> node that is streaming out thinks that the streaming session was successful
> {code}
> INFO [MiscStage:1] 2013-02-07 11:23:38,022 StreamOut.java (line 151) Stream context metadata [/data/cassandra/evidence/fingerprints/evidence-fingerprints-ia-472-Data.db sections=1727 progress=0/210208515 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db sections=1746 progress=0/119438030 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-920-Data.db sections=1681 progress=0/54498226 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-922-Data.db sections=16 progress=0/13490 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-918-Data.db sections=632 progress=0/70019542 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-921-Data.db sections=1644 progress=0/39870238 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-497-Data.db sections=1569 progress=0/208331077 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-923-Data.db sections=1572 progress=0/30870478 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-925-Data.db sections=167 progress=0/1845123 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-703-Data.db sections=1574 progress=0/287386471 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-913-Data.db sections=811 progress=0/103776521 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-915-Data.db sections=1539 progress=0/141864261 - 0%], 12 sstables.
> INFO [MiscStage:1] 2013-02-07 11:23:49,938 StreamOut.java (line 151) Stream context metadata [/var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db sections=3153 progress=0/994857654 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-918-Data.db sections=2507 progress=0/515992757 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-923-Data.db sections=3153 progress=0/131969347 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-703-Data.db sections=3068 progress=0/2784807967 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-913-Data.db sections=2696 progress=0/758409465 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ia-472-Data.db sections=3150 progress=0/1792868996 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-920-Data.db sections=3123 progress=0/240094510 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-929-Data.db sections=18 progress=0/29468 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-922-Data.db sections=217 progress=0/13490 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-921-Data.db sections=3124 progress=0/130320291 - 0%, /data/cassandra/evidence/fingerprints/evidence-fingerprints-ib-497-Data.db sections=3055 progress=0/1749539598 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-925-Data.db sections=1608 progress=0/13357410 - 0%, /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-915-Data.db sections=3096 progress=0/1241397203 - 0%], 13 sstables.
> INFO [Streaming to /10.8.25.132:2] 2013-02-07 11:24:18,780 StreamReplyVerbHandler.java (line 44) Successfully sent /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db to /10.8.25.132
> {code}
> node that is being streamed to has nothing in the logs about this particular sstable
> {code}
> Streaming from: /10.8.30.13
> evidence: /var/lib/cassandra/data2/evidence/fingerprints/evidence-fingerprints-ib-919-Data.db sections=3153 progress=218225400/994857654 - 21%
> _______
> Streaming from: /10.138.12.10
> evidence: /data2/cassandra/evidence/fingerprints/evidence-fingerprints-ib-26-Data.db sections=3153 progress=218225400/994857654 - 21%
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira