You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by xutom <xu...@126.com> on 2015/12/29 02:57:27 UTC
cassandra full gc too long
Hi all,
I have 5 nodes in my C* cluster, and each node has the same configuration file(Cassandra-env.sh: MAX_HEAP_SIZE="32G" and HEAP_NEWSIZE="8G"), and My Cassandra version is 2.1.1. Now I want to export all data of one table, i am using select * from tablename, and set the bStatement.setFetchSize(10000); When exporting the data, I check the status of the C* cluster, I found the following:
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 134.13.13.36 15.82 GB 256 ? 67aecea2-00f5-4001-ba4c-dc6138edaaa0 rack1
UN 134.13.13.37 20.25 GB 256 ? a8c0ec24-6e48-4082-be53-32a971a4aaaa rack1
DN 134.13.13.33 18.36 GB 256 ? d93f8201-a3a7-404a-8dd6-164ddbfeaaa7 rack1
UN 134.13.13.34 16.9 GB 256 ? 6880cf40-0b73-4bbc-a0a5-7799a983aaab rack1
UN 134.13.13.35 19.78 GB 256 ? 5658ae76-40a0-441f-908e-9d11ab2baaad rack1
The Cassandra jstat information on node 134.13.13.33 is following. I can see that Cassandra does not use S1, and it does not do the young GC, instead, it just do Full GC, and it do so many Full GC and so long, so C* cluster think it is Down. The node 134.13.13.33 can only work normally after startup for a while, after many times to do Full GC, it is Down and cannot finish the Full GC progress and crash at the end.
[cache@cache02 caslog]$ jstat -gcutil 62335 4000 2000
S0 S1 E O P YGC YGCT FGC FGCT GCT
86.49 0.00 100.00 98.02 72.54 19 38.236 3 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.52 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.52 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.52 19 38.236 4 0.863 39.099
0.00 0.00 41.13 100.00 72.52 19 38.236 4 71.671 109.906
0.00 0.00 41.13 100.00 72.52 19 38.236 4 71.671 109.906
0.00 0.00 41.13 100.00 72.52 19 38.236 4 71.671 109.906
0.00 0.00 41.13 100.00 72.52 19 38.236 4 71.671 109.906
0.00 0.00 77.65 100.00 60.03 19 38.236 5 71.671 109.906
20.27 0.00 100.00 100.00 60.27 19 38.236 5 73.905 112.141
58.59 0.00 100.00 100.00 60.30 19 38.236 5 73.905 112.141
98.19 0.00 100.00 100.00 60.30 19 38.236 5 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 5 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 5 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 5 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 5 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
0.00 0.00 78.90 100.00 60.30 19 38.236 6 141.773 180.008
0.00 0.00 78.90 100.00 60.30 19 38.236 6 141.773 180.008
0.00 0.00 78.90 100.00 60.30 19 38.236 6 141.773 180.008
Other Cassandra jstat information of other node of this C* cluster is following, I think it is normal because it can do young GC so many times then do Full GC:
[cache@cache06 ~]$ jstat -gcutil 32060 4000 2000
S0 S1 E O P YGC YGCT FGC FGCT GCT
0.00 100.00 26.53 59.08 60.02 609 140.372 12 89.422 229.794
0.00 100.00 33.48 61.20 60.02 611 140.834 12 89.422 230.257
0.00 100.00 50.18 63.71 60.02 613 141.384 12 89.422 230.807
11.65 0.00 83.51 66.31 60.02 616 141.723 12 89.422 231.146
100.00 100.00 100.00 68.40 60.02 619 142.167 12 89.422 231.590
0.00 100.00 13.15 71.60 60.02 621 143.195 12 89.422 232.618
0.00 100.00 27.69 73.89 60.02 623 143.679 12 89.422 233.102
100.00 0.00 71.57 74.60 60.02 624 143.772 12 89.422 233.194
0.00 94.92 88.87 74.97 60.02 625 143.838 12 89.422 233.260
100.00 100.00 100.00 76.37 60.02 627 143.942 13 89.577 233.519
0.00 100.00 78.16 76.32 60.02 627 144.027 14 90.273 234.300
0.00 100.00 31.22 58.51 60.02 629 144.425 14 90.273 234.698
100.00 0.00 27.90 44.76 60.02 632 145.104 14 90.273 235.377
100.00 0.00 32.10 29.26 60.02 634 145.254 14 90.273 235.527
83.50 0.00 65.04 27.60 59.78 636 145.395 14 90.273 235.669
100.00 0.00 72.60 28.20 59.78 638 145.534 14 90.273 235.807
0.00 100.00 5.97 28.73 59.78 641 145.703 14 90.273 235.976
0.00 76.34 34.28 29.32 59.78 643 145.877 14 90.273 236.150
0.00 97.42 91.21 29.50 59.79 645 145.971 14 90.273 236.244
0.00 100.00 94.70 29.72 59.79 647 146.070 14 90.273 236.343
78.65 0.00 46.82 30.19 59.79 650 146.201 14 90.273 236.475
59.25 0.00 49.98 30.19 59.79 652 146.267 14 90.273 236.540
Re: Re: cassandra full gc too long
Posted by Sebastian Estevez <se...@datastax.com>.
Hi Xutom,
What the " modern CQL client with paging support" mean? Is there opensource
> CQL client ? I does not use any opensource CQL client and exporting data
> with my java code.
Use the datastax Java driver which is open source:
github:
https://github.com/datastax/java-driver
docs:
http://datastax.github.io/java-driver/
I have splitted that table into 47*20=940 parts, I have 47 partitions and
> each partition also has 20 buckets, so everytime I execute such cql: select
> * from table where partition_id=a and bucket_id=b, and the number of each
> select result maybe 40-80 million.
I don't know what your data model looks like but it sounds (from your
description) that your partitions are too large. Check out my blog post on
data modeling and benchmarking:
http://www.sestevez.com/data-modeler/
Run stress on your test setup to gain confidence in your data model and to
ensure and benefit from predictable scalability.
All the best,
[image: datastax_logo.png] <http://www.datastax.com/>
Sebastián Estévez
Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
[image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
<https://twitter.com/datastax> [image: g+.png]
<https://plus.google.com/+Datastax/about>
<http://feeds.feedburner.com/datastax>
<http://goog_410786983>
<http://www.datastax.com/gartner-magic-quadrant-odbms>
DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.
On Tue, Dec 29, 2015 at 2:04 AM, xutom <xu...@126.com> wrote:
>
> Thanks for ur reply. Exactly I have splitted that table into 47*20=940
> parts, I have 47 partitions and each partition also has 20 buckets, so
> everytime I execute such cql: select * from table where partition_id=a and
> bucket_id=b, and the number of each select result maybe 40-80 million.
> What the " modern CQL client with paging support" mean? Is there
> opensource CQL client ? I does not use any opensource CQL client and
> exporting data with my java code.
>
>
> 在 2015-12-29 11:38:51，"Robert Coli" <rc...@eventbrite.com> 写道：
>
> On Mon, Dec 28, 2015 at 5:57 PM, xutom <xu...@126.com> wrote:
>
>> I have 5 nodes in my C* cluster, and each node has the same
>> configuration file(Cassandra-env.sh: MAX_HEAP_SIZE="32G" and
>> HEAP_NEWSIZE="8G"), and My Cassandra version is 2.1.1. Now I want to
>> export all data of one table, i am using select * from tablename,
>>
>
> Probably lower your heap size. If you're using CMS GC with 32gb heap you
> will get long GC pauses.
>
> Also use a modern CQL client with paging support.
>
> In addition, upgrade to the head of 2.1.x, 2.1.1 is not a version anyone
> should be using in production at this time.
>
> =Rob
>
>
>
>
>
Re:Re: cassandra full gc too long
Posted by xutom <xu...@126.com>.
Thanks for ur reply. Exactly I have splitted that table into 47*20=940 parts, I have 47 partitions and each partition also has 20 buckets, so everytime I execute such cql: select * from table where partition_id=a and bucket_id=b, and the number of each select result maybe 40-80 million.
What the " modern CQL client with paging support" mean? Is there opensource CQL client ? I does not use any opensource CQL client and exporting data with my java code.
在 2015-12-29 11:38:51，"Robert Coli" <rc...@eventbrite.com> 写道：
On Mon, Dec 28, 2015 at 5:57 PM, xutom <xu...@126.com> wrote:
I have 5 nodes in my C* cluster, and each node has the same configuration file(Cassandra-env.sh: MAX_HEAP_SIZE="32G" and HEAP_NEWSIZE="8G"), and My Cassandra version is 2.1.1. Now I want to export all data of one table, i am using select * from tablename,
Probably lower your heap size. If you're using CMS GC with 32gb heap you will get long GC pauses.
Also use a modern CQL client with paging support.
In addition, upgrade to the head of 2.1.x, 2.1.1 is not a version anyone should be using in production at this time.
=Rob
Re: cassandra full gc too long
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Dec 28, 2015 at 5:57 PM, xutom <xu...@126.com> wrote:
> I have 5 nodes in my C* cluster, and each node has the same
> configuration file(Cassandra-env.sh: MAX_HEAP_SIZE="32G" and
> HEAP_NEWSIZE="8G"), and My Cassandra version is 2.1.1. Now I want to
> export all data of one table, i am using select * from tablename,
>
Probably lower your heap size. If you're using CMS GC with 32gb heap you
will get long GC pauses.
Also use a modern CQL client with paging support.
In addition, upgrade to the head of 2.1.x, 2.1.1 is not a version anyone
should be using in production at this time.
=Rob