You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Ekaterina Dimitrova (Jira)" <ji...@apache.org> on 2021/11/03 15:48:00 UTC
[jira] [Comment Edited] (CASSANDRA-16349) SSTableLoader reports
error when SSTable(s) do not have data for some nodes
[ https://issues.apache.org/jira/browse/CASSANDRA-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17437675#comment-17437675 ]
Ekaterina Dimitrova edited comment on CASSANDRA-16349 at 11/3/21, 3:47 PM:
---------------------------------------------------------------------------
I think we need to mark the dtests to be since 3.0. Even if 2.2 is out of support, we still have it in CI and there might be third parties who still support it.
I can confirm that the test passes with 3.0 and 3.11 locally.
As apparently a release happened somewhere around this work, I will backport the patch to 4.0 branch too. Still don't have time for proper review so I will post whatever I have time for and leave it to NEEDS COMMITTER status in case someone has cycles before me.
[trunk|https://github.com/ekaterinadimitrova2/cassandra/pull/new/16349-streaming-sstableloader-4.0] | [dtest|https://github.com/ekaterinadimitrova2/cassandra-dtest/pull/new/CASSANDRA-16349] | [Jenkins CI |https://jenkins-cm4.apache.org/job/Cassandra-devbranch/1260/]
was (Author: e.dimitrova):
[trunk|https://github.com/ekaterinadimitrova2/cassandra/pull/new/16349-streaming-sstableloader-4.0] | [dtest|https://github.com/ekaterinadimitrova2/cassandra-dtest/pull/new/CASSANDRA-16349] | [Jenkins CI | https://jenkins-cm4.apache.org/job/Cassandra-devbranch/1260/]
> SSTableLoader reports error when SSTable(s) do not have data for some nodes
> ---------------------------------------------------------------------------
>
> Key: CASSANDRA-16349
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16349
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/sstable
> Reporter: Serban Teodorescu
> Assignee: Serban Teodorescu
> Priority: Normal
> Fix For: 4.0.x
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Running SSTableLoader in verbose mode will show error(s) if there are node(s) that do not own any data from the SSTable(s). This can happen in at least 2 cases:
> # SSTableLoader is used to stream backups while keeping the same token ranges
> # SSTable(s) are created with CQLSSTableWriter to match token ranges (this can bring better performance by using ZeroCopy streaming)
> Partial output of the SSTableLoader:
> {quote}ERROR 02:47:47,842 [Stream #fa8e73b0-3da5-11eb-9c47-c5d27ae8fe47] Remote peer /127.0.0.4:7000 failed stream session.
> ERROR 02:47:47,842 [Stream #fa8e73b0-3da5-11eb-9c47-c5d27ae8fe47] Remote peer /127.0.0.3:7000 failed stream session.
> progress: [/127.0.0.4:7000]0:0/1 100% [/127.0.0.3:7000]0:0/1 100% [/127.0.0.2:7000]0:7/7 100% [/127.0.0.1:7000]0:7/7 100% total: 100% 0.000KiB/s (avg: 1.611KiB/s)
> progress: [/127.0.0.4:7000]0:0/1 100% [/127.0.0.3:7000]0:0/1 100% [/127.0.0.2:7000]0:7/7 100% [/127.0.0.1:7000]0:7/7 100% total: 100% 0.000KiB/s (avg: 1.611KiB/s)
> progress: [/127.0.0.4:7000]0:0/1 100% [/127.0.0.3:7000]0:0/1 100% [/127.0.0.2:7000]0:7/7 100% [/127.0.0.1:7000]0:7/7 100% total: 100% 0.000KiB/s (avg: 1.515KiB/s)
> progress: [/127.0.0.4:7000]0:0/1 100% [/127.0.0.3:7000]0:0/1 100% [/127.0.0.2:7000]0:7/7 100% [/127.0.0.1:7000]0:7/7 100% total: 100% 0.000KiB/s (avg: 1.427KiB/s)
> {quote}
>
> Stack trace:
> {quote}java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
> at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:552)
> at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:533)
> at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:99)
> at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:49)
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88)
> at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1056)
> at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
> at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138)
> at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958)
> at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748)
> at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:220)
> at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:196)
> at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:505)
> at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:819)
> at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:595)
> at org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:189)
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:844)
> {quote}
> To reproduce create a cluster with ccm with more nodes than the RF, put some data into it copy a SSTable and stream it.
>
> The error originates on the nodes, the following stack trace is shown in the logs:
> {quote}java.lang.IllegalStateException: Stream hasn't been read yet
> at com.google.common.base.Preconditions.checkState(Preconditions.java:507)
> at org.apache.cassandra.db.streaming.CassandraIncomingFile.getSize(CassandraIncomingFile.java:96)
> at org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:789)
> at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:587)
> at org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:189)
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:844)
> {quote}
>
> An error is thrown due to stream size being read before any data was received. The solution would be not to stream this at all; SSTableLoader.java already looks into each SSTable to determine what parts of it will map to each node token ranges.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org