You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Yifan Cai (Jira)" <ji...@apache.org> on 2021/02/18 21:44:00 UTC
[jira] [Comment Edited] (CASSANDRA-16345) Test SSTables are in the
correct location after range movement with 1/5/20 data directories
[ https://issues.apache.org/jira/browse/CASSANDRA-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286735#comment-17286735 ]
Yifan Cai edited comment on CASSANDRA-16345 at 2/18/21, 9:43 PM:
-----------------------------------------------------------------
Ran the bootstrap and replacement test as suggested. All the tests run with live traffic.
When zero copy streaming (ZCS) is enabled, it can lead to the streamed files contain tokens that exceed the disk boundaries (only when using multiple data directories) on the target host. Therefore, running {{relocatesstables}} nodetool command can find misplaced files. After turning off zero copy streaming, the streamed files on the target node are correctly placed. We should document the side-effect of ZCS in the multiple data directories setup.
In the case of host replacement, all sstable files on the new host are correctly located.
*Scenario: STCS with 5 data directories. Bootstrap a new node. Zero copy streaming enabled.*
{code:java}
➜ grep -E 'JOINING|\[Stream #|Relocate|RELOCATE' cassandra-5d-relocatesstables-stcs-bootstrap-zcs-on.hlog
2021-02-13 04:40:17,316 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for ring information
2021-02-13 04:40:25,317 INFO [main] org.apache.cassandra.service.StorageService - JOINING: schema complete, ready to bootstrap
2021-02-13 04:40:25,317 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for pending range calculation
2021-02-13 04:40:25,317 INFO [main] org.apache.cassandra.service.StorageService - JOINING: calculation complete, ready to bootstrap
2021-02-13 04:40:25,317 INFO [main] org.apache.cassandra.service.StorageService - JOINING: getting bootstrap token
2021-02-13 04:40:25,381 INFO [main] org.apache.cassandra.service.StorageService - JOINING: sleeping 30000 ms for pending range setup
2021-02-13 04:40:55,382 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Starting to bootstrap...
2021-02-13 04:40:55,474 INFO [main] org.apache.cassandra.streaming.StreamResultFuture - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc] Executing streaming plan for Bootstrap
2021-02-13 04:40:55,478 INFO [main] org.apache.cassandra.streaming.StreamSession - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc] Starting streaming to /IP-1:7000
2021-02-13 04:40:55,498 INFO [main] org.apache.cassandra.streaming.StreamCoordinator - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc, ID#0] Beginning stream session with /IP-1:7000
2021-02-13 04:40:58,708 INFO [Stream-Deserializer-/IP-1:7000-bdeac312] org.apache.cassandra.streaming.StreamResultFuture - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc ID#0] Prepare completed. Receiving 76 files(53.205GiB), sending 0 files(0.000KiB)
2021-02-13 04:40:58,713 INFO [Stream-Deserializer-/IP-1:7000-bdeac312] org.apache.cassandra.streaming.StreamSession - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc] Starting streaming to /IP-2:7000
2021-02-13 04:40:58,723 INFO [Stream-Deserializer-/IP-1:7000-bdeac312] org.apache.cassandra.streaming.StreamCoordinator - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc, ID#0] Beginning stream session with /IP-2:7000
2021-02-13 04:41:00,725 INFO [Stream-Deserializer-/IP-2:7000-a982dcc1] org.apache.cassandra.streaming.StreamResultFuture - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc ID#0] Prepare completed. Receiving 94 files(53.242GiB), sending 0 files(0.000KiB)
2021-02-13 04:41:00,726 INFO [Stream-Deserializer-/IP-2:7000-a982dcc1] org.apache.cassandra.streaming.StreamSession - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc] Starting streaming to /IP-3:7000
2021-02-13 04:41:00,735 INFO [Stream-Deserializer-/IP-2:7000-a982dcc1] org.apache.cassandra.streaming.StreamCoordinator - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc, ID#0] Beginning stream session with /IP-3:7000
2021-02-13 04:41:11,252 INFO [Stream-Deserializer-/IP-3:7000-7be14fe5] org.apache.cassandra.streaming.StreamResultFuture - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc ID#0] Prepare completed. Receiving 135 files(53.276GiB), sending 0 files(0.000KiB)
2021-02-13 05:30:18,055 INFO [Stream-Deserializer-/IP-2:7000-a982dcc1] org.apache.cassandra.streaming.StreamResultFuture - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc] Session with /IP-2:7000 is complete
2021-02-13 05:31:57,310 INFO [Stream-Deserializer-/IP-1:7000-bdeac312] org.apache.cassandra.streaming.StreamResultFuture - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc] Session with /IP-1:7000 is complete
2021-02-13 05:42:05,457 INFO [Stream-Deserializer-/IP-3:7000-7be14fe5] org.apache.cassandra.streaming.StreamResultFuture - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc] Session with /IP-3:7000 is complete
2021-02-13 05:42:05,461 INFO [Stream-Deserializer-/IP-3:7000-7be14fe5] org.apache.cassandra.streaming.StreamResultFuture - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc] All sessions completed
2021-02-13 05:42:05,582 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Finish joining ring
2021-02-13 06:01:09,368 INFO [RMI TCP Connection(34)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate sstables to correct disk for tlp_stress.keyvaluelargeblob_2
2021-02-13 06:05:53,685 INFO [RMI TCP Connection(34)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - Finished Relocate sstables to correct disk for tlp_stress.keyvaluelargeblob_2 successfully
{code}
*Scenario: LCS with 5 data directories. Bootstrap a new node. Zero copy streaming enabled.*
{code:java}
➜ grep -E 'JOINING|\[Stream #|Relocate|RELOCATE' cassandra-5d-relocatesstables-lcs-bootstrap-zcs-on.hlog
2021-02-13 01:15:57,426 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for ring information
2021-02-13 01:16:05,428 INFO [main] org.apache.cassandra.service.StorageService - JOINING: schema complete, ready to bootstrap
2021-02-13 01:16:05,428 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for pending range calculation
2021-02-13 01:16:05,428 INFO [main] org.apache.cassandra.service.StorageService - JOINING: calculation complete, ready to bootstrap
2021-02-13 01:16:05,428 INFO [main] org.apache.cassandra.service.StorageService - JOINING: getting bootstrap token
2021-02-13 01:16:05,500 INFO [main] org.apache.cassandra.service.StorageService - JOINING: sleeping 30000 ms for pending range setup
2021-02-13 01:16:35,502 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Starting to bootstrap...
2021-02-13 01:16:35,606 INFO [main] org.apache.cassandra.streaming.StreamResultFuture - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a] Executing streaming plan for Bootstrap
2021-02-13 01:16:35,607 INFO [main] org.apache.cassandra.streaming.StreamSession - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a] Starting streaming to /IP-1:7000
2021-02-13 01:16:35,631 INFO [main] org.apache.cassandra.streaming.StreamCoordinator - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a, ID#0] Beginning stream session with /IP-1:7000
2021-02-13 01:16:42,438 INFO [Stream-Deserializer-/IP-1:7000-e3172a2e] org.apache.cassandra.streaming.StreamResultFuture - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a ID#0] Prepare completed. Receiving 2866 files(51.911GiB), sending 0 files(0.000KiB)
2021-02-13 01:16:42,441 INFO [Stream-Deserializer-/IP-1:7000-e3172a2e] org.apache.cassandra.streaming.StreamSession - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a] Starting streaming to /IP-2:7000
2021-02-13 01:16:42,449 INFO [Stream-Deserializer-/IP-1:7000-e3172a2e] org.apache.cassandra.streaming.StreamCoordinator - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a, ID#0] Beginning stream session with /IP-2:7000
2021-02-13 01:16:48,052 INFO [Stream-Deserializer-/IP-2:7000-8a158dcb] org.apache.cassandra.streaming.StreamResultFuture - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a ID#0] Prepare completed. Receiving 2452 files(45.599GiB), sending 0 files(0.000KiB)
2021-02-13 01:16:48,053 INFO [Stream-Deserializer-/IP-2:7000-8a158dcb] org.apache.cassandra.streaming.StreamSession - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a] Starting streaming to /IP-3:7000
2021-02-13 01:16:48,062 INFO [Stream-Deserializer-/IP-2:7000-8a158dcb] org.apache.cassandra.streaming.StreamCoordinator - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a, ID#0] Beginning stream session with /IP-3:7000
2021-02-13 01:16:53,664 INFO [Stream-Deserializer-/IP-3:7000-6e3aacd4] org.apache.cassandra.streaming.StreamResultFuture - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a ID#0] Prepare completed. Receiving 2850 files(51.402GiB), sending 0 files(0.000KiB)
2021-02-13 01:48:11,733 INFO [Stream-Deserializer-/IP-2:7000-8a158dcb] org.apache.cassandra.streaming.StreamResultFuture - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a] Session with /IP-2:7000 is complete
2021-02-13 01:52:19,583 INFO [Stream-Deserializer-/IP-3:7000-6e3aacd4] org.apache.cassandra.streaming.StreamResultFuture - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a] Session with /IP-3:7000 is complete
2021-02-13 01:52:31,993 INFO [Stream-Deserializer-/IP-1:7000-e3172a2e] org.apache.cassandra.streaming.StreamResultFuture - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a] Session with /IP-1:7000 is complete
2021-02-13 01:52:31,997 INFO [Stream-Deserializer-/IP-1:7000-e3172a2e] org.apache.cassandra.streaming.StreamResultFuture - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a] All sessions completed
2021-02-13 02:12:32,685 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Finish joining ring
2021-02-13 04:05:19,396 INFO [RMI TCP Connection(16)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate sstables to correct disk for tlp_stress.keyvaluelargeblob_2
2021-02-13 04:05:27,915 INFO [RMI TCP Connection(16)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - Finished Relocate sstables to correct disk for tlp_stress.keyvaluelargeblob_2 successfully
{code}
*Scenario: STCS with 5 data directories. Bootstrap a new node. Zero copy streaming disabled.*
{code:java}
➜ grep -E 'JOINING|\[Stream #|Relocate|RELOCATE' cassandra-5d-relocatesstables-stcs-bootstrap-zcs-off.hlog
2021-02-17 22:00:54,579 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for ring information
2021-02-17 22:01:03,581 INFO [main] org.apache.cassandra.service.StorageService - JOINING: schema complete, ready to bootstrap
2021-02-17 22:01:03,581 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for pending range calculation
2021-02-17 22:01:03,581 INFO [main] org.apache.cassandra.service.StorageService - JOINING: calculation complete, ready to bootstrap
2021-02-17 22:01:03,581 INFO [main] org.apache.cassandra.service.StorageService - JOINING: getting bootstrap token
2021-02-17 22:01:03,605 INFO [main] org.apache.cassandra.service.StorageService - JOINING: sleeping 30000 ms for pending range setup
2021-02-17 22:01:33,606 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Starting to bootstrap...
2021-02-17 22:01:33,694 INFO [main] org.apache.cassandra.streaming.StreamResultFuture - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e] Executing streaming plan for Bootstrap
2021-02-17 22:01:33,700 INFO [main] org.apache.cassandra.streaming.StreamSession - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e] Starting streaming to /IP-1:7000
2021-02-17 22:01:33,725 INFO [main] org.apache.cassandra.streaming.StreamCoordinator - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e, ID#0] Beginning stream session with /IP-1:7000
2021-02-17 22:01:40,131 INFO [Stream-Deserializer-/IP-1:7000-4c1b63b2] org.apache.cassandra.streaming.StreamResultFuture - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e ID#0] Prepare completed. Receiving 44 files(51.447GiB), sending 0 files(0.000KiB)
2021-02-17 22:01:40,135 INFO [Stream-Deserializer-/IP-1:7000-4c1b63b2] org.apache.cassandra.streaming.StreamSession - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e] Starting streaming to /IP-2:7000
2021-02-17 22:01:40,143 INFO [Stream-Deserializer-/IP-1:7000-4c1b63b2] org.apache.cassandra.streaming.StreamCoordinator - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e, ID#0] Beginning stream session with /IP-2:7000
2021-02-17 22:01:46,946 INFO [Stream-Deserializer-/IP-2:7000-117220e5] org.apache.cassandra.streaming.StreamResultFuture - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e ID#0] Prepare completed. Receiving 41 files(51.389GiB), sending 0 files(0.000KiB)
2021-02-17 22:01:46,947 INFO [Stream-Deserializer-/IP-2:7000-117220e5] org.apache.cassandra.streaming.StreamSession - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e] Starting streaming to /IP-3:7000
2021-02-17 22:01:46,957 INFO [Stream-Deserializer-/IP-2:7000-117220e5] org.apache.cassandra.streaming.StreamCoordinator - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e, ID#0] Beginning stream session with /IP-3:7000
2021-02-17 22:01:50,958 INFO [Stream-Deserializer-/IP-3:7000-7f9975a8] org.apache.cassandra.streaming.StreamResultFuture - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e ID#0] Prepare completed. Receiving 36 files(51.365GiB), sending 0 files(0.000KiB)
2021-02-17 22:37:04,025 INFO [Stream-Deserializer-/IP-1:7000-4c1b63b2] org.apache.cassandra.streaming.StreamResultFuture - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e] Session with /IP-1:7000 is complete
2021-02-17 22:37:11,633 INFO [Stream-Deserializer-/IP-3:7000-7f9975a8] org.apache.cassandra.streaming.StreamResultFuture - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e] Session with /IP-3:7000 is complete
2021-02-17 22:37:21,252 INFO [Stream-Deserializer-/IP-2:7000-117220e5] org.apache.cassandra.streaming.StreamResultFuture - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e] Session with /IP-2:7000 is complete
2021-02-17 22:37:21,254 INFO [Stream-Deserializer-/IP-2:7000-117220e5] org.apache.cassandra.streaming.StreamResultFuture - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e] All sessions completed
2021-02-17 22:37:22,284 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Finish joining ring
2021-02-18 03:03:25,463 INFO [RMI TCP Connection(64)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate sstables to correct disk for tlp_stress.keyvaluelargeblob_2
2021-02-18 03:03:25,466 INFO [RMI TCP Connection(64)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - No sstables to RELOCATE for tlp_stress.keyvaluelargeblob_2
{code}
*Scenario: STCS with 5 data directories. Bootstrap a new node. Zero copy streaming disabled.*
{code:java}
➜ grep -E 'JOINING|\[Stream #|Relocate|RELOCATE' cassandra-5d-relocatesstables-lcs-bootstrap-zcs-off.hlog
2021-02-17 22:04:24,399 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for ring information
2021-02-17 22:04:34,402 INFO [main] org.apache.cassandra.service.StorageService - JOINING: schema complete, ready to bootstrap
2021-02-17 22:04:34,402 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for pending range calculation
2021-02-17 22:04:34,402 INFO [main] org.apache.cassandra.service.StorageService - JOINING: calculation complete, ready to bootstrap
2021-02-17 22:04:34,402 INFO [main] org.apache.cassandra.service.StorageService - JOINING: getting bootstrap token
2021-02-17 22:04:34,469 INFO [main] org.apache.cassandra.service.StorageService - JOINING: sleeping 30000 ms for pending range setup
2021-02-17 22:05:04,470 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Starting to bootstrap...
2021-02-17 22:05:04,549 INFO [main] org.apache.cassandra.streaming.StreamResultFuture - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212] Executing streaming plan for Bootstrap
2021-02-17 22:05:04,550 INFO [main] org.apache.cassandra.streaming.StreamSession - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212] Starting streaming to /IP-1:7000
2021-02-17 22:05:04,620 INFO [main] org.apache.cassandra.streaming.StreamCoordinator - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212, ID#0] Beginning stream session with /IP-1:7000
2021-02-17 22:05:07,827 INFO [Stream-Deserializer-/IP-1:7000-1b75a614] org.apache.cassandra.streaming.StreamResultFuture - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212 ID#0] Prepare completed. Receiving 438 files(51.316GiB), sending 0 files(0.000KiB)
2021-02-17 22:05:07,830 INFO [Stream-Deserializer-/IP-1:7000-1b75a614] org.apache.cassandra.streaming.StreamSession - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212] Starting streaming to /IP-2:7000
2021-02-17 22:05:08,042 INFO [Stream-Deserializer-/IP-1:7000-1b75a614] org.apache.cassandra.streaming.StreamCoordinator - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212, ID#0] Beginning stream session with /IP-2:7000
2021-02-17 22:05:12,445 INFO [Stream-Deserializer-/IP-2:7000-7520c488] org.apache.cassandra.streaming.StreamResultFuture - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212 ID#0] Prepare completed. Receiving 434 files(51.314GiB), sending 0 files(0.000KiB)
2021-02-17 22:05:12,446 INFO [Stream-Deserializer-/IP-2:7000-7520c488] org.apache.cassandra.streaming.StreamSession - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212] Starting streaming to /IP-3:7000
2021-02-17 22:05:12,456 INFO [Stream-Deserializer-/IP-2:7000-7520c488] org.apache.cassandra.streaming.StreamCoordinator - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212, ID#0] Beginning stream session with /IP-3:7000
2021-02-17 22:05:17,019 INFO [Stream-Deserializer-/IP-3:7000-ce13e3ad] org.apache.cassandra.streaming.StreamResultFuture - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212 ID#0] Prepare completed. Receiving 423 files(51.299GiB), sending 0 files(0.000KiB)
2021-02-17 22:40:21,049 INFO [Stream-Deserializer-/IP-1:7000-1b75a614] org.apache.cassandra.streaming.StreamResultFuture - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212] Session with /IP-1:7000 is complete
2021-02-17 22:40:21,974 INFO [Stream-Deserializer-/IP-3:7000-ce13e3ad] org.apache.cassandra.streaming.StreamResultFuture - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212] Session with /IP-3:7000 is complete
2021-02-17 22:40:27,173 INFO [Stream-Deserializer-/IP-2:7000-7520c488] org.apache.cassandra.streaming.StreamResultFuture - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212] Session with /IP-2:7000 is complete
2021-02-17 22:40:27,175 INFO [Stream-Deserializer-/IP-2:7000-7520c488] org.apache.cassandra.streaming.StreamResultFuture - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212] All sessions completed
2021-02-17 22:40:27,250 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Finish joining ring
2021-02-18 03:03:17,796 INFO [RMI TCP Connection(31)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate sstables to correct disk for tlp_stress.keyvaluelargeblob_2
2021-02-18 03:03:18,067 INFO [RMI TCP Connection(31)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - No sstables to RELOCATE for tlp_stress.keyvaluelargeblob_2
{code}
*Scenario: STCS with 5 data directories. Replace a node.*
{code:java}
➜ grep -E 'JOINING|\[Stream #|Relocate|RELOCATE' cassandra-5d-relocatesstables-stcs-replacement.hlog
2021-02-11 02:21:00,896 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for ring information
2021-02-11 02:21:10,898 INFO [main] org.apache.cassandra.service.StorageService - JOINING: schema complete, ready to bootstrap
2021-02-11 02:21:10,898 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for pending range calculation
2021-02-11 02:21:10,898 INFO [main] org.apache.cassandra.service.StorageService - JOINING: calculation complete, ready to bootstrap
2021-02-11 02:22:10,899 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Replacing a node with token(s): [6148914691236517205]
2021-02-11 02:22:10,926 INFO [main] org.apache.cassandra.service.StorageService - JOINING: sleeping 30000 ms for pending range setup
2021-02-11 02:22:40,928 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Starting to bootstrap...
2021-02-11 02:22:41,021 INFO [main] org.apache.cassandra.streaming.StreamResultFuture - [Stream #042014e0-6c10-11eb-a301-13a607fa169d] Executing streaming plan for Bootstrap
2021-02-11 02:22:41,022 INFO [main] org.apache.cassandra.streaming.StreamSession - [Stream #042014e0-6c10-11eb-a301-13a607fa169d] Starting streaming to /IP-1:7000
2021-02-11 02:22:41,041 INFO [main] org.apache.cassandra.streaming.StreamCoordinator - [Stream #042014e0-6c10-11eb-a301-13a607fa169d, ID#0] Beginning stream session with /IP-1:7000
2021-02-11 02:23:07,052 INFO [Stream-Deserializer-/IP-1:7000-ee139048] org.apache.cassandra.streaming.StreamResultFuture - [Stream #042014e0-6c10-11eb-a301-13a607fa169d ID#0] Prepare completed. Receiving 201 files(68.648GiB), sending 0 files(0.000KiB)
2021-02-11 02:23:07,055 INFO [Stream-Deserializer-/IP-1:7000-ee139048] org.apache.cassandra.streaming.StreamSession - [Stream #042014e0-6c10-11eb-a301-13a607fa169d] Starting streaming to /IP-2:7000
2021-02-11 02:23:07,073 INFO [Stream-Deserializer-/IP-1:7000-ee139048] org.apache.cassandra.streaming.StreamCoordinator - [Stream #042014e0-6c10-11eb-a301-13a607fa169d, ID#0] Beginning stream session with /IP-2:7000
2021-02-11 02:23:20,678 INFO [Stream-Deserializer-/IP-2:7000-30717ccc] org.apache.cassandra.streaming.StreamResultFuture - [Stream #042014e0-6c10-11eb-a301-13a607fa169d ID#0] Prepare completed. Receiving 381 files(137.270GiB), sending 0 files(0.000KiB)
2021-02-11 03:15:56,727 INFO [Stream-Deserializer-/IP-1:7000-ee139048] org.apache.cassandra.streaming.StreamResultFuture - [Stream #042014e0-6c10-11eb-a301-13a607fa169d] Session with /IP-1:7000 is complete
2021-02-11 04:38:44,316 INFO [Stream-Deserializer-/IP-2:7000-30717ccc] org.apache.cassandra.streaming.StreamResultFuture - [Stream #042014e0-6c10-11eb-a301-13a607fa169d] Session with /IP-2:7000 is complete
2021-02-11 04:38:44,319 INFO [Stream-Deserializer-/IP-2:7000-30717ccc] org.apache.cassandra.streaming.StreamResultFuture - [Stream #042014e0-6c10-11eb-a301-13a607fa169d] All sessions completed
2021-02-11 04:38:44,374 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Finish joining ring
2021-02-11 05:05:53,442 INFO [RMI TCP Connection(304)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate sstables to correct disk for tlp_stress.keyvaluelargeblob_2
2021-02-11 05:05:53,445 INFO [RMI TCP Connection(304)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - No sstables to RELOCATE for tlp_stress.keyvaluelargeblob_2
{code}
*Scenario: LCS with 5 data directories. Replace a node.*
{code:java}
➜ grep -E 'JOINING|\[Stream #|Relocate|RELOCATE' cassandra-5d-relocatesstables-lcs-replacement.hlog
2021-02-11 05:37:52,764 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for ring information
2021-02-11 05:38:03,766 INFO [main] org.apache.cassandra.service.StorageService - JOINING: schema complete, ready to bootstrap
2021-02-11 05:38:03,766 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for pending range calculation
2021-02-11 05:38:03,766 INFO [main] org.apache.cassandra.service.StorageService - JOINING: calculation complete, ready to bootstrap
2021-02-11 05:39:03,767 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Replacing a node with token(s): [6148914691236517205]
2021-02-11 05:39:03,797 INFO [main] org.apache.cassandra.service.StorageService - JOINING: sleeping 30000 ms for pending range setup
2021-02-11 05:39:33,798 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Starting to bootstrap...
2021-02-11 05:39:33,887 INFO [main] org.apache.cassandra.streaming.StreamResultFuture - [Stream #85246300-6c2b-11eb-a88a-852966917da0] Executing streaming plan for Bootstrap
2021-02-11 05:39:33,888 INFO [main] org.apache.cassandra.streaming.StreamSession - [Stream #85246300-6c2b-11eb-a88a-852966917da0] Starting streaming to /IP-1:7000
2021-02-11 05:39:33,909 INFO [main] org.apache.cassandra.streaming.StreamCoordinator - [Stream #85246300-6c2b-11eb-a88a-852966917da0, ID#0] Beginning stream session with /IP-1:7000
2021-02-11 05:40:59,562 INFO [Stream-Deserializer-/IP-1:7000-d215bef1] org.apache.cassandra.streaming.StreamResultFuture - [Stream #85246300-6c2b-11eb-a88a-852966917da0 ID#0] Prepare completed. Receiving 8014 files(137.307GiB), sending 0 files(0.000KiB)
2021-02-11 05:40:59,563 INFO [Stream-Deserializer-/IP-1:7000-d215bef1] org.apache.cassandra.streaming.StreamSession - [Stream #85246300-6c2b-11eb-a88a-852966917da0] Starting streaming to /IP-2:7000
2021-02-11 05:40:59,576 INFO [Stream-Deserializer-/IP-1:7000-d215bef1] org.apache.cassandra.streaming.StreamCoordinator - [Stream #85246300-6c2b-11eb-a88a-852966917da0, ID#0] Beginning stream session with /IP-2:7000
2021-02-11 05:41:39,586 INFO [Stream-Deserializer-/IP-2:7000-76d3889b] org.apache.cassandra.streaming.StreamResultFuture - [Stream #85246300-6c2b-11eb-a88a-852966917da0 ID#0] Prepare completed. Receiving 3844 files(68.687GiB), sending 0 files(0.000KiB)
2021-02-11 06:49:20,756 INFO [Stream-Deserializer-/IP-2:7000-76d3889b] org.apache.cassandra.streaming.StreamResultFuture - [Stream #85246300-6c2b-11eb-a88a-852966917da0] Session with /IP-2:7000 is complete
2021-02-11 07:42:38,365 INFO [Stream-Deserializer-/IP-1:7000-d215bef1] org.apache.cassandra.streaming.StreamResultFuture - [Stream #85246300-6c2b-11eb-a88a-852966917da0] Session with /IP-1:7000 is complete
2021-02-11 07:42:38,370 INFO [Stream-Deserializer-/IP-1:7000-d215bef1] org.apache.cassandra.streaming.StreamResultFuture - [Stream #85246300-6c2b-11eb-a88a-852966917da0] All sessions completed
2021-02-11 07:42:38,447 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Finish joining ring
2021-02-11 07:53:45,575 INFO [RMI TCP Connection(267)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate sstables to correct disk for tlp_stress.keyvaluelargeblob_2
2021-02-11 07:53:45,937 INFO [RMI TCP Connection(267)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - No sstables to RELOCATE for tlp_stress.keyvaluelargeblob_2
{code}
was (Author: yifanc):
Ran the bootstrap and replacement test as suggested. All the tests run with live traffic.
When zero copy streaming (ZCS) is enabled, it can lead to the streamed files contain tokens that exceed the disk boundaries (only when using multiple data directories) on the target host. Therefore, running {{relocatesstables}} nodetool command can find misplaced files. After turning off zero copy streaming, the streamed files on the target node are correctly placed. We should document the side-effect of ZCS in the multiple data directories setup.
In the case of host replacement, all sstable files on the new host are correctly located.
*Scenario: STCS with 5 data directories. Bootstrap a new node. Zero copy streaming enabled.*
{code:java}
➜ grep -E 'JOINING|\[Stream #|Relocate|RELOCATE' cassandra-5d-relocatesstables-stcs-bootstrap-zcs-on.hlog
2021-02-13 04:40:17,316 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for ring information
2021-02-13 04:40:25,317 INFO [main] org.apache.cassandra.service.StorageService - JOINING: schema complete, ready to bootstrap
2021-02-13 04:40:25,317 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for pending range calculation
2021-02-13 04:40:25,317 INFO [main] org.apache.cassandra.service.StorageService - JOINING: calculation complete, ready to bootstrap
2021-02-13 04:40:25,317 INFO [main] org.apache.cassandra.service.StorageService - JOINING: getting bootstrap token
2021-02-13 04:40:25,381 INFO [main] org.apache.cassandra.service.StorageService - JOINING: sleeping 30000 ms for pending range setup
2021-02-13 04:40:55,382 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Starting to bootstrap...
2021-02-13 04:40:55,474 INFO [main] org.apache.cassandra.streaming.StreamResultFuture - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc] Executing streaming plan for Bootstrap
2021-02-13 04:40:55,478 INFO [main] org.apache.cassandra.streaming.StreamSession - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc] Starting streaming to /IP-1:7000
2021-02-13 04:40:55,498 INFO [main] org.apache.cassandra.streaming.StreamCoordinator - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc, ID#0] Beginning stream session with /IP-1:7000
2021-02-13 04:40:58,708 INFO [Stream-Deserializer-/IP-1:7000-bdeac312] org.apache.cassandra.streaming.StreamResultFuture - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc ID#0] Prepare completed. Receiving 76 files(53.205GiB), sending 0 files(0.000KiB)
2021-02-13 04:40:58,713 INFO [Stream-Deserializer-/IP-1:7000-bdeac312] org.apache.cassandra.streaming.StreamSession - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc] Starting streaming to /IP-2:7000
2021-02-13 04:40:58,723 INFO [Stream-Deserializer-/IP-1:7000-bdeac312] org.apache.cassandra.streaming.StreamCoordinator - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc, ID#0] Beginning stream session with /IP-2:7000
2021-02-13 04:41:00,725 INFO [Stream-Deserializer-/IP-2:7000-a982dcc1] org.apache.cassandra.streaming.StreamResultFuture - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc ID#0] Prepare completed. Receiving 94 files(53.242GiB), sending 0 files(0.000KiB)
2021-02-13 04:41:00,726 INFO [Stream-Deserializer-/IP-2:7000-a982dcc1] org.apache.cassandra.streaming.StreamSession - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc] Starting streaming to /IP-3:7000
2021-02-13 04:41:00,735 INFO [Stream-Deserializer-/IP-2:7000-a982dcc1] org.apache.cassandra.streaming.StreamCoordinator - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc, ID#0] Beginning stream session with /IP-3:7000
2021-02-13 04:41:11,252 INFO [Stream-Deserializer-/IP-3:7000-7be14fe5] org.apache.cassandra.streaming.StreamResultFuture - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc ID#0] Prepare completed. Receiving 135 files(53.276GiB), sending 0 files(0.000KiB)
2021-02-13 05:30:18,055 INFO [Stream-Deserializer-/IP-2:7000-a982dcc1] org.apache.cassandra.streaming.StreamResultFuture - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc] Session with /IP-2:7000 is complete
2021-02-13 05:31:57,310 INFO [Stream-Deserializer-/IP-1:7000-bdeac312] org.apache.cassandra.streaming.StreamResultFuture - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc] Session with /IP-1:7000 is complete
2021-02-13 05:42:05,457 INFO [Stream-Deserializer-/IP-3:7000-7be14fe5] org.apache.cassandra.streaming.StreamResultFuture - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc] Session with /IP-3:7000 is complete
2021-02-13 05:42:05,461 INFO [Stream-Deserializer-/IP-3:7000-7be14fe5] org.apache.cassandra.streaming.StreamResultFuture - [Stream #a8d3c0d0-6db5-11eb-a794-e31cdd68c1bc] All sessions completed
2021-02-13 05:42:05,582 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Finish joining ring
2021-02-13 06:01:09,368 INFO [RMI TCP Connection(34)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate sstables to correct disk for tlp_stress.keyvaluelargeblob_2
2021-02-13 06:05:53,685 INFO [RMI TCP Connection(34)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - Finished Relocate sstables to correct disk for tlp_stress.keyvaluelargeblob_2 successfully
{code}
*Scenario: LCS with 5 data directories. Bootstrap a new node. Zero copy streaming enabled.*
{code:java}
➜ grep -E 'JOINING|\[Stream #|Relocate|RELOCATE' cassandra-5d-relocatesstables-lcs-bootstrap-zcs-on.hlog
2021-02-13 01:15:57,426 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for ring information
2021-02-13 01:16:05,428 INFO [main] org.apache.cassandra.service.StorageService - JOINING: schema complete, ready to bootstrap
2021-02-13 01:16:05,428 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for pending range calculation
2021-02-13 01:16:05,428 INFO [main] org.apache.cassandra.service.StorageService - JOINING: calculation complete, ready to bootstrap
2021-02-13 01:16:05,428 INFO [main] org.apache.cassandra.service.StorageService - JOINING: getting bootstrap token
2021-02-13 01:16:05,500 INFO [main] org.apache.cassandra.service.StorageService - JOINING: sleeping 30000 ms for pending range setup
2021-02-13 01:16:35,502 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Starting to bootstrap...
2021-02-13 01:16:35,606 INFO [main] org.apache.cassandra.streaming.StreamResultFuture - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a] Executing streaming plan for Bootstrap
2021-02-13 01:16:35,607 INFO [main] org.apache.cassandra.streaming.StreamSession - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a] Starting streaming to /IP-1:7000
2021-02-13 01:16:35,631 INFO [main] org.apache.cassandra.streaming.StreamCoordinator - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a, ID#0] Beginning stream session with /IP-1:7000
2021-02-13 01:16:42,438 INFO [Stream-Deserializer-/IP-1:7000-e3172a2e] org.apache.cassandra.streaming.StreamResultFuture - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a ID#0] Prepare completed. Receiving 2866 files(51.911GiB), sending 0 files(0.000KiB)
2021-02-13 01:16:42,441 INFO [Stream-Deserializer-/IP-1:7000-e3172a2e] org.apache.cassandra.streaming.StreamSession - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a] Starting streaming to /IP-2:7000
2021-02-13 01:16:42,449 INFO [Stream-Deserializer-/IP-1:7000-e3172a2e] org.apache.cassandra.streaming.StreamCoordinator - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a, ID#0] Beginning stream session with /IP-2:7000
2021-02-13 01:16:48,052 INFO [Stream-Deserializer-/IP-2:7000-8a158dcb] org.apache.cassandra.streaming.StreamResultFuture - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a ID#0] Prepare completed. Receiving 2452 files(45.599GiB), sending 0 files(0.000KiB)
2021-02-13 01:16:48,053 INFO [Stream-Deserializer-/IP-2:7000-8a158dcb] org.apache.cassandra.streaming.StreamSession - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a] Starting streaming to /IP-3:7000
2021-02-13 01:16:48,062 INFO [Stream-Deserializer-/IP-2:7000-8a158dcb] org.apache.cassandra.streaming.StreamCoordinator - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a, ID#0] Beginning stream session with /IP-3:7000
2021-02-13 01:16:53,664 INFO [Stream-Deserializer-/IP-3:7000-6e3aacd4] org.apache.cassandra.streaming.StreamResultFuture - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a ID#0] Prepare completed. Receiving 2850 files(51.402GiB), sending 0 files(0.000KiB)
2021-02-13 01:48:11,733 INFO [Stream-Deserializer-/IP-2:7000-8a158dcb] org.apache.cassandra.streaming.StreamResultFuture - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a] Session with /IP-2:7000 is complete
2021-02-13 01:52:19,583 INFO [Stream-Deserializer-/IP-3:7000-6e3aacd4] org.apache.cassandra.streaming.StreamResultFuture - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a] Session with /IP-3:7000 is complete
2021-02-13 01:52:31,993 INFO [Stream-Deserializer-/IP-1:7000-e3172a2e] org.apache.cassandra.streaming.StreamResultFuture - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a] Session with /IP-1:7000 is complete
2021-02-13 01:52:31,997 INFO [Stream-Deserializer-/IP-1:7000-e3172a2e] org.apache.cassandra.streaming.StreamResultFuture - [Stream #1d5f2290-6d99-11eb-b66b-b96624b1bb4a] All sessions completed
2021-02-13 02:12:32,685 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Finish joining ring
2021-02-13 04:05:19,396 INFO [RMI TCP Connection(16)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate sstables to correct disk for tlp_stress.keyvaluelargeblob_2
2021-02-13 04:05:27,915 INFO [RMI TCP Connection(16)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - Finished Relocate sstables to correct disk for tlp_stress.keyvaluelargeblob_2 successfully
{code}
*Scenario: STCS with 5 data directories. Bootstrap a new node. Zero copy streaming disabled.*
{code:java}
➜ grep -E 'JOINING|\[Stream #|Relocate|RELOCATE' cassandra-5d-relocatesstables-stcs-bootstrap-zcs-off.hlog
2021-02-17 22:00:54,579 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for ring information
2021-02-17 22:01:03,581 INFO [main] org.apache.cassandra.service.StorageService - JOINING: schema complete, ready to bootstrap
2021-02-17 22:01:03,581 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for pending range calculation
2021-02-17 22:01:03,581 INFO [main] org.apache.cassandra.service.StorageService - JOINING: calculation complete, ready to bootstrap
2021-02-17 22:01:03,581 INFO [main] org.apache.cassandra.service.StorageService - JOINING: getting bootstrap token
2021-02-17 22:01:03,605 INFO [main] org.apache.cassandra.service.StorageService - JOINING: sleeping 30000 ms for pending range setup
2021-02-17 22:01:33,606 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Starting to bootstrap...
2021-02-17 22:01:33,694 INFO [main] org.apache.cassandra.streaming.StreamResultFuture - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e] Executing streaming plan for Bootstrap
2021-02-17 22:01:33,700 INFO [main] org.apache.cassandra.streaming.StreamSession - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e] Starting streaming to /IP-1:7000
2021-02-17 22:01:33,725 INFO [main] org.apache.cassandra.streaming.StreamCoordinator - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e, ID#0] Beginning stream session with /IP-1:7000
2021-02-17 22:01:40,131 INFO [Stream-Deserializer-/IP-1:7000-4c1b63b2] org.apache.cassandra.streaming.StreamResultFuture - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e ID#0] Prepare completed. Receiving 44 files(51.447GiB), sending 0 files(0.000KiB)
2021-02-17 22:01:40,135 INFO [Stream-Deserializer-/IP-1:7000-4c1b63b2] org.apache.cassandra.streaming.StreamSession - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e] Starting streaming to /IP-2:7000
2021-02-17 22:01:40,143 INFO [Stream-Deserializer-/IP-1:7000-4c1b63b2] org.apache.cassandra.streaming.StreamCoordinator - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e, ID#0] Beginning stream session with /IP-2:7000
2021-02-17 22:01:46,946 INFO [Stream-Deserializer-/IP-2:7000-117220e5] org.apache.cassandra.streaming.StreamResultFuture - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e ID#0] Prepare completed. Receiving 41 files(51.389GiB), sending 0 files(0.000KiB)
2021-02-17 22:01:46,947 INFO [Stream-Deserializer-/IP-2:7000-117220e5] org.apache.cassandra.streaming.StreamSession - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e] Starting streaming to /IP-3:7000
2021-02-17 22:01:46,957 INFO [Stream-Deserializer-/IP-2:7000-117220e5] org.apache.cassandra.streaming.StreamCoordinator - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e, ID#0] Beginning stream session with /IP-3:7000
2021-02-17 22:01:50,958 INFO [Stream-Deserializer-/IP-3:7000-7f9975a8] org.apache.cassandra.streaming.StreamResultFuture - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e ID#0] Prepare completed. Receiving 36 files(51.365GiB), sending 0 files(0.000KiB)
2021-02-17 22:37:04,025 INFO [Stream-Deserializer-/IP-1:7000-4c1b63b2] org.apache.cassandra.streaming.StreamResultFuture - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e] Session with /IP-1:7000 is complete
2021-02-17 22:37:11,633 INFO [Stream-Deserializer-/IP-3:7000-7f9975a8] org.apache.cassandra.streaming.StreamResultFuture - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e] Session with /IP-3:7000 is complete
2021-02-17 22:37:21,252 INFO [Stream-Deserializer-/IP-2:7000-117220e5] org.apache.cassandra.streaming.StreamResultFuture - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e] Session with /IP-2:7000 is complete
2021-02-17 22:37:21,254 INFO [Stream-Deserializer-/IP-2:7000-117220e5] org.apache.cassandra.streaming.StreamResultFuture - [Stream #b2901b00-716b-11eb-afc9-23042398cf4e] All sessions completed
2021-02-17 22:37:22,284 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Finish joining ring
2021-02-18 03:03:25,463 INFO [RMI TCP Connection(64)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate sstables to correct disk for tlp_stress.keyvaluelargeblob_2
2021-02-18 03:03:25,466 INFO [RMI TCP Connection(64)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - No sstables to RELOCATE for tlp_stress.keyvaluelargeblob_2
{code}
*Scenario: STCS with 5 data directories. Bootstrap a new node. Zero copy streaming disabled.*
{code:java}
➜ grep -E 'JOINING|\[Stream #|Relocate|RELOCATE' cassandra-5d-relocatesstables-lcs-bootstrap-zcs-off.hlog
2021-02-17 22:04:24,399 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for ring information
2021-02-17 22:04:34,402 INFO [main] org.apache.cassandra.service.StorageService - JOINING: schema complete, ready to bootstrap
2021-02-17 22:04:34,402 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for pending range calculation
2021-02-17 22:04:34,402 INFO [main] org.apache.cassandra.service.StorageService - JOINING: calculation complete, ready to bootstrap
2021-02-17 22:04:34,402 INFO [main] org.apache.cassandra.service.StorageService - JOINING: getting bootstrap token
2021-02-17 22:04:34,469 INFO [main] org.apache.cassandra.service.StorageService - JOINING: sleeping 30000 ms for pending range setup
2021-02-17 22:05:04,470 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Starting to bootstrap...
2021-02-17 22:05:04,549 INFO [main] org.apache.cassandra.streaming.StreamResultFuture - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212] Executing streaming plan for Bootstrap
2021-02-17 22:05:04,550 INFO [main] org.apache.cassandra.streaming.StreamSession - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212] Starting streaming to /IP-1:7000
2021-02-17 22:05:04,620 INFO [main] org.apache.cassandra.streaming.StreamCoordinator - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212, ID#0] Beginning stream session with /IP-1:7000
2021-02-17 22:05:07,827 INFO [Stream-Deserializer-/IP-1:7000-1b75a614] org.apache.cassandra.streaming.StreamResultFuture - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212 ID#0] Prepare completed. Receiving 438 files(51.316GiB), sending 0 files(0.000KiB)
2021-02-17 22:05:07,830 INFO [Stream-Deserializer-/IP-1:7000-1b75a614] org.apache.cassandra.streaming.StreamSession - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212] Starting streaming to /IP-2:7000
2021-02-17 22:05:08,042 INFO [Stream-Deserializer-/IP-1:7000-1b75a614] org.apache.cassandra.streaming.StreamCoordinator - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212, ID#0] Beginning stream session with /IP-2:7000
2021-02-17 22:05:12,445 INFO [Stream-Deserializer-/IP-2:7000-7520c488] org.apache.cassandra.streaming.StreamResultFuture - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212 ID#0] Prepare completed. Receiving 434 files(51.314GiB), sending 0 files(0.000KiB)
2021-02-17 22:05:12,446 INFO [Stream-Deserializer-/IP-2:7000-7520c488] org.apache.cassandra.streaming.StreamSession - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212] Starting streaming to /IP-3:7000
2021-02-17 22:05:12,456 INFO [Stream-Deserializer-/IP-2:7000-7520c488] org.apache.cassandra.streaming.StreamCoordinator - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212, ID#0] Beginning stream session with /IP-3:7000
2021-02-17 22:05:17,019 INFO [Stream-Deserializer-/IP-3:7000-ce13e3ad] org.apache.cassandra.streaming.StreamResultFuture - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212 ID#0] Prepare completed. Receiving 423 files(51.299GiB), sending 0 files(0.000KiB)
2021-02-17 22:40:21,049 INFO [Stream-Deserializer-/IP-1:7000-1b75a614] org.apache.cassandra.streaming.StreamResultFuture - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212] Session with /IP-1:7000 is complete
2021-02-17 22:40:21,974 INFO [Stream-Deserializer-/IP-3:7000-ce13e3ad] org.apache.cassandra.streaming.StreamResultFuture - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212] Session with /IP-3:7000 is complete
2021-02-17 22:40:27,173 INFO [Stream-Deserializer-/IP-2:7000-7520c488] org.apache.cassandra.streaming.StreamResultFuture - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212] Session with /IP-2:7000 is complete
2021-02-17 22:40:27,175 INFO [Stream-Deserializer-/IP-2:7000-7520c488] org.apache.cassandra.streaming.StreamResultFuture - [Stream #303f17e0-716c-11eb-81ab-85ed6e921212] All sessions completed
2021-02-17 22:40:27,250 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Finish joining ring
2021-02-18 03:03:17,796 INFO [RMI TCP Connection(31)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate sstables to correct disk for tlp_stress.keyvaluelargeblob_2
2021-02-18 03:03:18,067 INFO [RMI TCP Connection(31)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - No sstables to RELOCATE for tlp_stress.keyvaluelargeblob_2
{code}
*Scenario: STCS with 5 data directories. Replace a node. *
{code:java}
➜ grep -E 'JOINING|\[Stream #|Relocate|RELOCATE' cassandra-5d-relocatesstables-stcs-replacement.hlog
2021-02-11 02:21:00,896 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for ring information
2021-02-11 02:21:10,898 INFO [main] org.apache.cassandra.service.StorageService - JOINING: schema complete, ready to bootstrap
2021-02-11 02:21:10,898 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for pending range calculation
2021-02-11 02:21:10,898 INFO [main] org.apache.cassandra.service.StorageService - JOINING: calculation complete, ready to bootstrap
2021-02-11 02:22:10,899 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Replacing a node with token(s): [6148914691236517205]
2021-02-11 02:22:10,926 INFO [main] org.apache.cassandra.service.StorageService - JOINING: sleeping 30000 ms for pending range setup
2021-02-11 02:22:40,928 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Starting to bootstrap...
2021-02-11 02:22:41,021 INFO [main] org.apache.cassandra.streaming.StreamResultFuture - [Stream #042014e0-6c10-11eb-a301-13a607fa169d] Executing streaming plan for Bootstrap
2021-02-11 02:22:41,022 INFO [main] org.apache.cassandra.streaming.StreamSession - [Stream #042014e0-6c10-11eb-a301-13a607fa169d] Starting streaming to /IP-1:7000
2021-02-11 02:22:41,041 INFO [main] org.apache.cassandra.streaming.StreamCoordinator - [Stream #042014e0-6c10-11eb-a301-13a607fa169d, ID#0] Beginning stream session with /IP-1:7000
2021-02-11 02:23:07,052 INFO [Stream-Deserializer-/IP-1:7000-ee139048] org.apache.cassandra.streaming.StreamResultFuture - [Stream #042014e0-6c10-11eb-a301-13a607fa169d ID#0] Prepare completed. Receiving 201 files(68.648GiB), sending 0 files(0.000KiB)
2021-02-11 02:23:07,055 INFO [Stream-Deserializer-/IP-1:7000-ee139048] org.apache.cassandra.streaming.StreamSession - [Stream #042014e0-6c10-11eb-a301-13a607fa169d] Starting streaming to /IP-2:7000
2021-02-11 02:23:07,073 INFO [Stream-Deserializer-/IP-1:7000-ee139048] org.apache.cassandra.streaming.StreamCoordinator - [Stream #042014e0-6c10-11eb-a301-13a607fa169d, ID#0] Beginning stream session with /IP-2:7000
2021-02-11 02:23:20,678 INFO [Stream-Deserializer-/IP-2:7000-30717ccc] org.apache.cassandra.streaming.StreamResultFuture - [Stream #042014e0-6c10-11eb-a301-13a607fa169d ID#0] Prepare completed. Receiving 381 files(137.270GiB), sending 0 files(0.000KiB)
2021-02-11 03:15:56,727 INFO [Stream-Deserializer-/IP-1:7000-ee139048] org.apache.cassandra.streaming.StreamResultFuture - [Stream #042014e0-6c10-11eb-a301-13a607fa169d] Session with /IP-1:7000 is complete
2021-02-11 04:38:44,316 INFO [Stream-Deserializer-/IP-2:7000-30717ccc] org.apache.cassandra.streaming.StreamResultFuture - [Stream #042014e0-6c10-11eb-a301-13a607fa169d] Session with /IP-2:7000 is complete
2021-02-11 04:38:44,319 INFO [Stream-Deserializer-/IP-2:7000-30717ccc] org.apache.cassandra.streaming.StreamResultFuture - [Stream #042014e0-6c10-11eb-a301-13a607fa169d] All sessions completed
2021-02-11 04:38:44,374 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Finish joining ring
2021-02-11 05:05:53,442 INFO [RMI TCP Connection(304)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate sstables to correct disk for tlp_stress.keyvaluelargeblob_2
2021-02-11 05:05:53,445 INFO [RMI TCP Connection(304)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - No sstables to RELOCATE for tlp_stress.keyvaluelargeblob_2
{code}
*Scenario: LCS with 5 data directories. Replace a node.*
{code:java}
➜ grep -E 'JOINING|\[Stream #|Relocate|RELOCATE' cassandra-5d-relocatesstables-lcs-replacement.hlog
2021-02-11 05:37:52,764 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for ring information
2021-02-11 05:38:03,766 INFO [main] org.apache.cassandra.service.StorageService - JOINING: schema complete, ready to bootstrap
2021-02-11 05:38:03,766 INFO [main] org.apache.cassandra.service.StorageService - JOINING: waiting for pending range calculation
2021-02-11 05:38:03,766 INFO [main] org.apache.cassandra.service.StorageService - JOINING: calculation complete, ready to bootstrap
2021-02-11 05:39:03,767 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Replacing a node with token(s): [6148914691236517205]
2021-02-11 05:39:03,797 INFO [main] org.apache.cassandra.service.StorageService - JOINING: sleeping 30000 ms for pending range setup
2021-02-11 05:39:33,798 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Starting to bootstrap...
2021-02-11 05:39:33,887 INFO [main] org.apache.cassandra.streaming.StreamResultFuture - [Stream #85246300-6c2b-11eb-a88a-852966917da0] Executing streaming plan for Bootstrap
2021-02-11 05:39:33,888 INFO [main] org.apache.cassandra.streaming.StreamSession - [Stream #85246300-6c2b-11eb-a88a-852966917da0] Starting streaming to /IP-1:7000
2021-02-11 05:39:33,909 INFO [main] org.apache.cassandra.streaming.StreamCoordinator - [Stream #85246300-6c2b-11eb-a88a-852966917da0, ID#0] Beginning stream session with /IP-1:7000
2021-02-11 05:40:59,562 INFO [Stream-Deserializer-/IP-1:7000-d215bef1] org.apache.cassandra.streaming.StreamResultFuture - [Stream #85246300-6c2b-11eb-a88a-852966917da0 ID#0] Prepare completed. Receiving 8014 files(137.307GiB), sending 0 files(0.000KiB)
2021-02-11 05:40:59,563 INFO [Stream-Deserializer-/IP-1:7000-d215bef1] org.apache.cassandra.streaming.StreamSession - [Stream #85246300-6c2b-11eb-a88a-852966917da0] Starting streaming to /IP-2:7000
2021-02-11 05:40:59,576 INFO [Stream-Deserializer-/IP-1:7000-d215bef1] org.apache.cassandra.streaming.StreamCoordinator - [Stream #85246300-6c2b-11eb-a88a-852966917da0, ID#0] Beginning stream session with /IP-2:7000
2021-02-11 05:41:39,586 INFO [Stream-Deserializer-/IP-2:7000-76d3889b] org.apache.cassandra.streaming.StreamResultFuture - [Stream #85246300-6c2b-11eb-a88a-852966917da0 ID#0] Prepare completed. Receiving 3844 files(68.687GiB), sending 0 files(0.000KiB)
2021-02-11 06:49:20,756 INFO [Stream-Deserializer-/IP-2:7000-76d3889b] org.apache.cassandra.streaming.StreamResultFuture - [Stream #85246300-6c2b-11eb-a88a-852966917da0] Session with /IP-2:7000 is complete
2021-02-11 07:42:38,365 INFO [Stream-Deserializer-/IP-1:7000-d215bef1] org.apache.cassandra.streaming.StreamResultFuture - [Stream #85246300-6c2b-11eb-a88a-852966917da0] Session with /IP-1:7000 is complete
2021-02-11 07:42:38,370 INFO [Stream-Deserializer-/IP-1:7000-d215bef1] org.apache.cassandra.streaming.StreamResultFuture - [Stream #85246300-6c2b-11eb-a88a-852966917da0] All sessions completed
2021-02-11 07:42:38,447 INFO [main] org.apache.cassandra.service.StorageService - JOINING: Finish joining ring
2021-02-11 07:53:45,575 INFO [RMI TCP Connection(267)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - Starting Relocate sstables to correct disk for tlp_stress.keyvaluelargeblob_2
2021-02-11 07:53:45,937 INFO [RMI TCP Connection(267)-127.0.0.1] org.apache.cassandra.db.compaction.CompactionManager - No sstables to RELOCATE for tlp_stress.keyvaluelargeblob_2
{code}
> Test SSTables are in the correct location after range movement with 1/5/20 data directories
> -------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16345
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16345
> Project: Cassandra
> Issue Type: Sub-task
> Components: Test/benchmark
> Reporter: Yifan Cai
> Assignee: Yifan Cai
> Priority: Normal
>
> The testing cluster should be pre-populated with ~200GB data in each node. Change the token ranges and verify that each data directory only contains SSTables belongs to it. In other word, no SSTables are relocated by running nodetool "relocatesstables". Both LCS and STCS need to be covered.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org