You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by xutom <xu...@126.com> on 2016/01/04 15:52:40 UTC
Fail to export all data in C* cluster
Hi all,
I have a C* cluster with 6 nodes. My cassandra version is 2.1.1. I start 50 threads to insert datas into C* cluster, each thread inserts about up to 100 million rows with the same partition key. After inserting all the datas, I start another app with 50 threads to export all the datas into localfile, I using such cqlsh: select * from table where partition_id=xxx(each partition has about 100 million rows). But unfortunately I fail to export all the datas: I run 3 times, and each time I get the different number of results. If I successfully export all datas, everytime I should get the same number of results, is it right?
Best Regards
Re: Re: Fail to export all data in C* cluster
Posted by Jack Krupansky <ja...@gmail.com>.
Cassandra does not have any automatic rollback of partial work for a failed
write request, if with CL=ALL - that just assures that you will get an
error indication if not all the writes succeed. You do need to keep
retrying failed writes until they succeed. Otherwise the partial work will
still be in the database until you run repair. Note my step 1 specified the
need for retries.
If you are getting just a handful of failures and retries, that's probably
okay, but if you are getting tons of failures then some manual throttling
(sleep between requests) may be the better approach, but even then your app
should be prepared for occasional failures, like due to Java GC or network
latency or whatever.
If you can do the retries at CL=ALL until they succeed, then you should be
set and not need repair.
And as I set in approach #3, if you throttle back enough then you can
probably avoid the need for CL=ALL.
-- Jack Krupansky
On Tue, Jan 5, 2016 at 2:43 AM, xutom <xu...@126.com> wrote:
> Dear Jack,
> Thanks!
> My keyspace is such as:
> test@cqlsh> DESC KEYSPACE sky ;
> CREATE KEYSPACE sky WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '3'} AND durable_writes = true;
> CREATE TABLE sky.user1 (pati int, uuid text, name text, name2 text,
> PRIMARY KEY (pati, uuid))
>
> Now I am using CL=ALL during inserting and set the retry policy by
> such following codes:
> RetryPolicy rp = new CustomRetryPolicy(3, 3, 2);
> Cluster cluster =
> Cluster.builder().addContactPoint(seedIp).withCredentials(
> "test", "test")
> .withRetryPolicy(rp)
> .withLoadBalancingPolicy(
> new TokenAwarePolicy(new
> DCAwareRoundRobinPolicy())) /*Here My Cluster has 6 nodes, all in the
> same DataCenter*/
> .build();
> PreparedStatement insertStatement = session
> .prepare("INSERT INTO " + tableName
> + "(" + columns + ") "
> + "VALUES (?, ?, ?, ?);");
> insertStatement.setConsistencyLevel(ConsistencyLevel.ALL); /* Here I set
> the CL to ALL */
> I start 30 threads to insert datas, each thread uses BatchStatement to
> insert 100 rows with the same partition key but different primary key every
> time, and run 100000 times. After Inserting about 99084500 rows into C*
> cluster with many timeout exceptions I stop the inserting progress and then
> use following codes to export all the datas into local file:
> String cqlstr = " select * from " + this.tableName
> + " where pati = " + this.partitionssss[i];
> PreparedStatement Statement = session
> .prepare(cqlstr);
> BoundStatement bStatement = new BoundStatement(
> Statement);
> bStatement.setFetchSize(10000);
> iter = session.execute(bStatement).iterator();
> then I write the results( in iter) to localfile. I run 3 times and all
> three results are different。
>
> I have set the CL to ALL when inserting datas, but why I get the
> different results when I export all datas everytime?
> By the way, I have set : "hinted_handoff_enabled: true", is it the
> problem when the C* cluster is overloaded even if I have set the CL to ALL?
>
> Best Regrads
> jerry
>
> At 2016-01-04 23:37:20, "Jack Krupansky" <ja...@gmail.com> wrote:
>
> You have three choices:
>
> 1. Insert with CL=ALL, with client-level retries if the write fails due to
> the cluster being overloaded.
> 2. Insert with CL=QUORUM and then run repair after all data has been
> inserted.
> 3. Lower your insert rate in your client so that the cluster can keep up
> with your inserts.
>
> Yes, Cassandra supports eventual consistency, but if you overload the
> cluster, the hinted handoff for nodes beyond the requested CL may timeout
> and be discarded, hence the need for repair.
>
> What CL are you currently using for inserts?
>
>
> -- Jack Krupansky
>
> On Mon, Jan 4, 2016 at 9:52 AM, xutom <xu...@126.com> wrote:
>
>> Hi all,
>>
>> I have a C* cluster with 6 nodes. My cassandra version is 2.1.1. I
>> start 50 threads to insert datas into C* cluster, each thread inserts about
>> up to 100 million rows with the same partition key. After inserting all the
>> datas, I start another app with 50 threads to export all the datas into
>> localfile, I using such cqlsh: select * from table where
>> partition_id=xxx(each partition has about 100 million rows). But unfortunately
>> I fail to export all the datas: I run 3 times, and each time I get the
>> different number of results. If I successfully export all datas, everytime
>> I should get the same number of results, is it right?
>>
>> Best Regards
>>
>>
>>
>>
>
>
>
>
>
Re:Re: Fail to export all data in C* cluster
Posted by xutom <xu...@126.com>.
Dear Jack,
Thanks!
My keyspace is such as:
test@cqlsh> DESC KEYSPACE sky ;
CREATE KEYSPACE sky WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'} AND durable_writes = true;
CREATE TABLE sky.user1 (pati int, uuid text, name text, name2 text,
PRIMARY KEY (pati, uuid))
Now I am using CL=ALL during inserting and set the retry policy by such following codes:
RetryPolicy rp = new CustomRetryPolicy(3, 3, 2);
Cluster cluster = Cluster.builder().addContactPoint(seedIp).withCredentials(
"test", "test")
.withRetryPolicy(rp)
.withLoadBalancingPolicy(
new TokenAwarePolicy(new DCAwareRoundRobinPolicy())) /*Here My Cluster has 6 nodes, all in the same DataCenter*/
.build();
PreparedStatement insertStatement = session
.prepare("INSERT INTO " + tableName
+ "(" + columns + ") "
+ "VALUES (?, ?, ?, ?);");
insertStatement.setConsistencyLevel(ConsistencyLevel.ALL); /* Here I set the CL to ALL */
I start 30 threads to insert datas, each thread uses BatchStatement to insert 100 rows with the same partition key but different primary key every time, and run 100000 times. After Inserting about 99084500 rows into C* cluster with many timeout exceptions I stop the inserting progress and then use following codes to export all the datas into local file:
String cqlstr = " select * from " + this.tableName
+ " where pati = " + this.partitionssss[i];
PreparedStatement Statement = session
.prepare(cqlstr);
BoundStatement bStatement = new BoundStatement(
Statement);
bStatement.setFetchSize(10000);
iter = session.execute(bStatement).iterator();
then I write the results( in iter) to localfile. I run 3 times and all three results are different。
I have set the CL to ALL when inserting datas, but why I get the different results when I export all datas everytime?
By the way, I have set : "hinted_handoff_enabled: true", is it the problem when the C* cluster is overloaded even if I have set the CL to ALL?
Best Regrads
jerry
At 2016-01-04 23:37:20, "Jack Krupansky" <ja...@gmail.com> wrote:
You have three choices:
1. Insert with CL=ALL, with client-level retries if the write fails due to the cluster being overloaded.
2. Insert with CL=QUORUM and then run repair after all data has been inserted.
3. Lower your insert rate in your client so that the cluster can keep up with your inserts.
Yes, Cassandra supports eventual consistency, but if you overload the cluster, the hinted handoff for nodes beyond the requested CL may timeout and be discarded, hence the need for repair.
What CL are you currently using for inserts?
-- Jack Krupansky
On Mon, Jan 4, 2016 at 9:52 AM, xutom <xu...@126.com> wrote:
Hi all,
I have a C* cluster with 6 nodes. My cassandra version is 2.1.1. I start 50 threads to insert datas into C* cluster, each thread inserts about up to 100 million rows with the same partition key. After inserting all the datas, I start another app with 50 threads to export all the datas into localfile, I using such cqlsh: select * from table where partition_id=xxx(each partition has about 100 million rows). But unfortunately I fail to export all the datas: I run 3 times, and each time I get the different number of results. If I successfully export all datas, everytime I should get the same number of results, is it right?
Best Regards
Re: Fail to export all data in C* cluster
Posted by Jack Krupansky <ja...@gmail.com>.
You have three choices:
1. Insert with CL=ALL, with client-level retries if the write fails due to
the cluster being overloaded.
2. Insert with CL=QUORUM and then run repair after all data has been
inserted.
3. Lower your insert rate in your client so that the cluster can keep up
with your inserts.
Yes, Cassandra supports eventual consistency, but if you overload the
cluster, the hinted handoff for nodes beyond the requested CL may timeout
and be discarded, hence the need for repair.
What CL are you currently using for inserts?
-- Jack Krupansky
On Mon, Jan 4, 2016 at 9:52 AM, xutom <xu...@126.com> wrote:
> Hi all,
>
> I have a C* cluster with 6 nodes. My cassandra version is 2.1.1. I
> start 50 threads to insert datas into C* cluster, each thread inserts about
> up to 100 million rows with the same partition key. After inserting all the
> datas, I start another app with 50 threads to export all the datas into
> localfile, I using such cqlsh: select * from table where
> partition_id=xxx(each partition has about 100 million rows). But unfortunately
> I fail to export all the datas: I run 3 times, and each time I get the
> different number of results. If I successfully export all datas, everytime
> I should get the same number of results, is it right?
>
> Best Regards
>
>
>
>