You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jacek Furmankiewicz (JIRA)" <ji...@apache.org> on 2014/06/06 18:14:03 UTC
[jira] [Commented] (CASSANDRA-7361) Cassandra locks up in full GC,
unable to free memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019976#comment-14019976 ]
Jacek Furmankiewicz commented on CASSANDRA-7361:
------------------------------------------------
I have a tar.gz of the heapdump (2.3 GB). I am not able to attach it to JIRA. Let me know if there is anywhere we could upload it for you.
> Cassandra locks up in full GC, unable to free memory
> ----------------------------------------------------
>
> Key: CASSANDRA-7361
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7361
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu, RedHat, JDK 1.7
> Reporter: Jacek Furmankiewicz
> Priority: Blocker
>
> We have a long running batch load process, which runs for many hours.
> Massive amount of writes, in large mutation batches (we increase the thrift frame size to 45 MB).
> Everything goes well, but after about 3 hrs of processing everything locks up. We start getting NoHostsAvailable exceptions on the Java application side (with Astyanax as our driver), eventually socket timeouts.
> Looking at Cassandra, we can see that it is using nearly the full 8GB of heap and unable to free it. It spends most of its time in full GC, but the amount of memory does not go down.
> Here is a long sample from jstat to show this over an extended time period
> e.g.
> {quote}
> -bash-4.1$ $JAVA_HOME/bin/jstat -gc 3728 5s
> S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
> 81920.0 81920.0 0.0 4097.0 655360.0 150419.0 7569408.0 7202990.3 43280.0 25855.8 5271 268.378 4940 9877.726 10146.104
> 81920.0 81920.0 0.0 4097.0 655360.0 150657.2 7569408.0 7202990.3 43280.0 25855.8 5271 268.378 4940 9877.726 10146.104
> 81920.0 81920.0 0.0 4097.0 655360.0 150708.5 7569408.0 7202990.3 43280.0 25855.8 5271 268.378 4941 9877.950 10146.328
> 81920.0 81920.0 0.0 4097.0 655360.0 150920.4 7569408.0 7202990.2 43280.0 25855.8 5271 268.378 4942 9878.085 10146.463
> 81920.0 81920.0 0.0 4097.0 655360.0 150930.7 7569408.0 7202990.2 43280.0 25855.8 5271 268.378 4942 9878.085 10146.463
> 81920.0 81920.0 0.0 4097.0 655360.0 150998.2 7569408.0 7202990.2 43280.0 25855.8 5271 268.378 4942 9878.085 10146.463
> 81920.0 81920.0 0.0 4097.0 655360.0 151335.6 7569408.0 7202990.2 43280.0 25855.8 5271 268.378 4943 9878.253 10146.630
> 81920.0 81920.0 0.0 4097.0 655360.0 151483.3 7569408.0 7202990.2 43280.0 25855.8 5271 268.378 4944 9878.389 10146.766
> 81920.0 81920.0 0.0 4097.0 655360.0 151750.4 7569408.0 7202990.2 43280.0 25855.8 5271 268.378 4944 9878.389 10146.766
> 81920.0 81920.0 0.0 4097.0 655360.0 151752.5 7569408.0 7202990.2 43280.0 25855.8 5271 268.378 4944 9878.389 10146.766
> 81920.0 81920.0 0.0 4097.0 655360.0 151997.4 7569408.0 7202990.2 43280.0 25855.8 5271 268.378 4945 9878.593 10146.970
> 81920.0 81920.0 0.0 4097.0 655360.0 152063.0 7569408.0 7202988.6 43280.0 25855.8 5271 268.378 4946 9878.732 10147.110
> 81920.0 81920.0 0.0 4097.0 655360.0 152256.2 7569408.0 7202988.6 43280.0 25855.8 5271 268.378 4946 9878.732 10147.110
> 81920.0 81920.0 0.0 4097.0 655360.0 152276.2 7569408.0 7202988.6 43280.0 25855.8 5271 268.378 4946 9878.732 10147.110
> 81920.0 81920.0 0.0 4097.0 655360.0 152276.2 7569408.0 7202988.6 43280.0 25855.8 5271 268.378 4947 9878.968 10147.345
> 81920.0 81920.0 0.0 4097.0 655360.0 152525.3 7569408.0 7202857.5 43280.0 25855.8 5271 268.378 4948 9879.107 10147.484
> 81920.0 81920.0 0.0 4097.0 655360.0 152673.0 7569408.0 7202857.5 43280.0 25855.8 5271 268.378 4948 9879.107 10147.484
> 81920.0 81920.0 0.0 4097.0 655360.0 152855.5 7569408.0 7202857.5 43280.0 25855.8 5271 268.378 4948 9879.107 10147.484
> 81920.0 81920.0 0.0 4097.0 655360.0 152871.3 7569408.0 7202857.5 43280.0 25855.8 5271 268.378 4949 9879.275 10147.653
> 81920.0 81920.0 0.0 4097.0 655360.0 153116.2 7569408.0 7202857.0 43280.0 25855.8 5271 268.378 4950 9879.415 10147.793
> 81920.0 81920.0 0.0 4097.0 655360.0 153191.6 7569408.0 7202857.0 43280.0 25855.8 5271 268.378 4950 9879.415 10147.793
> 81920.0 81920.0 0.0 4097.0 655360.0 153454.3 7569408.0 7202857.0 43280.0 25855.8 5271 268.378 4950 9879.415 10147.793
> 81920.0 81920.0 0.0 4097.0 655360.0 153528.0 7569408.0 7202857.0 43280.0 25855.8 5271 268.378 4951 9879.583 10147.961
> 81920.0 81920.0 0.0 4097.0 655360.0 153766.1 7569408.0 7202857.0 43280.0 25855.8 5271 268.378 4952 9879.721 10148.099
> 81920.0 81920.0 0.0 4097.0 655360.0 153791.9 7569408.0 7202857.0 43280.0 25855.8 5271 268.378 4952 9879.721 10148.099
> 81920.0 81920.0 0.0 4097.0 655360.0 153855.1 7569408.0 7202857.0 43280.0 25855.8 5271 268.378 4952 9879.721 10148.099
> 81920.0 81920.0 0.0 4097.0 655360.0 154114.8 7569408.0 7202857.0 43280.0 25855.8 5271 268.378 4953 9879.953 10148.331
> 81920.0 81920.0 0.0 4097.0 655360.0 154118.8 7569408.0 7202856.8 43280.0 25855.8 5271 268.378 4954 9880.097 10148.474
> 81920.0 81920.0 0.0 4097.0 655360.0 154382.7 7569408.0 7202856.8 43280.0 25855.8 5271 268.378 4954 9880.097 10148.474
> 81920.0 81920.0 0.0 4097.0 655360.0 154511.8 7569408.0 7202856.8 43280.0 25855.8 5271 268.378 4954 9880.097 10148.474
> 81920.0 81920.0 0.0 4097.0 655360.0 154699.2 7569408.0 7202856.8 43280.0 25855.8 5271 268.378 4955 9880.273 10148.650
> 81920.0 81920.0 0.0 4097.0 655360.0 154707.4 7569408.0 7202856.7 43280.0 25855.8 5271 268.378 4956 9880.273 10148.650
> 81920.0 81920.0 0.0 4097.0 655360.0 154985.2 7569408.0 7202856.7 43280.0 25855.8 5271 268.378 4956 9880.414 10148.792
> 81920.0 81920.0 0.0 4097.0 655360.0 155006.1 7569408.0 7202856.7 43280.0 25855.8 5271 268.378 4956 9880.414 10148.792
> 81920.0 81920.0 0.0 4097.0 655360.0 155050.8 7569408.0 7202856.7 43280.0 25855.8 5271 268.378 4957 9880.585 10148.963
> 81920.0 81920.0 0.0 4097.0 655360.0 155233.1 7569408.0 7202856.7 43280.0 25855.8 5271 268.378 4957 9880.585 10148.963
> 81920.0 81920.0 0.0 4097.0 655360.0 155536.7 7569408.0 7202856.7 43280.0 25855.8 5271 268.378 4958 9880.730 10149.108
> 81920.0 81920.0 0.0 4097.0 655360.0 156764.1 7569408.0 7202856.7 43280.0 25855.8 5271 268.378 4958 9880.730 10149.108
> 81920.0 81920.0 0.0 4097.0 655360.0 156829.3 7569408.0 7202856.7 43280.0 25855.8 5271 268.378 4959 9880.963 10149.341
> 81920.0 81920.0 0.0 4097.0 655360.0 157015.2 7569408.0 7202856.6 43280.0 25855.8 5271 268.378 4959 9880.963 10149.341
> 81920.0 81920.0 0.0 4097.0 655360.0 157089.1 7569408.0 7202856.6 43280.0 25855.8 5271 268.378 4960 9881.104 10149.481
> 81920.0 81920.0 0.0 4097.0 655360.0 157353.1 7569408.0 7202856.6 43280.0 25855.8 5271 268.378 4960 9881.104 10149.481
> 81920.0 81920.0 0.0 4097.0 655360.0 157368.8 7569408.0 7202856.6 43280.0 25855.8 5271 268.378 4961 9881.279 10149.656
> 81920.0 81920.0 0.0 4097.0 655360.0 157608.5 7569408.0 7202856.6 43280.0 25855.8 5271 268.378 4961 9881.279 10149.656
> 81920.0 81920.0 0.0 4097.0 655360.0 157684.2 7569408.0 7202856.6 43280.0 25855.8 5271 268.378 4962 9881.422 10149.800
> 81920.0 81920.0 0.0 4097.0 655360.0 157922.4 7569408.0 7202856.6 43280.0 25855.8 5271 268.378 4962 9881.422 10149.800
> 81920.0 81920.0 0.0 4097.0 655360.0 157959.9 7569408.0 7202856.6 43280.0 25855.8 5271 268.378 4963 9881.600 10149.977
> 81920.0 81920.0 0.0 4097.0 655360.0 158004.8 7569408.0 7202856.5 43280.0 25855.8 5271 268.378 4963 9881.600 10149.977
> 81920.0 81920.0 0.0 4097.0 655360.0 158197.6 7569408.0 7202856.5 43280.0 25855.8 5271 268.378 4964 9881.742 10150.120
> 81920.0 81920.0 0.0 4097.0 655360.0 158209.8 7569408.0 7202856.5 43280.0 25855.8 5271 268.378 4964 9881.742 10150.120
> 81920.0 81920.0 0.0 4097.0 655360.0 158473.8 7569408.0 7202856.5 43280.0 25855.8 5271 268.378 4965 9881.920 10150.297
> 81920.0 81920.0 0.0 4097.0 655360.0 158473.8 7569408.0 7202856.5 43280.0 25855.8 5271 268.378 4965 9881.920 10150.297
> 81920.0 81920.0 0.0 4097.0 655360.0 158769.7 7569408.0 7202856.5 43280.0 25855.8 5271 268.378 4966 9882.063 10150.441
> 81920.0 81920.0 0.0 4097.0 655360.0 158796.6 7569408.0 7202856.5 43280.0 25855.8 5271 268.378 4966 9882.063 10150.441
> 81920.0 81920.0 0.0 4097.0 655360.0 159076.4 7569408.0 7202856.5 43280.0 25855.8 5271 268.378 4966 9882.063 10150.441
> 81920.0 81920.0 0.0 4097.0 655360.0 159080.7 7569408.0 7202856.4 43280.0 25855.8 5271 268.378 4967 9882.250 10150.628
> 81920.0 81920.0 0.0 4097.0 655360.0 159190.8 7569408.0 7202856.4 43280.0 25855.8 5271 268.378 4968 9882.394 10150.772
> 81920.0 81920.0 0.0 4097.0 655360.0 159391.8 7569408.0 7202856.4 43280.0 25855.8 5271 268.378 4968 9882.394 10150.772
> 81920.0 81920.0 0.0 4097.0 655360.0 159697.5 7569408.0 7202856.4 43280.0 25855.8 5271 268.378 4968 9882.394 10150.772
> 81920.0 81920.0 0.0 4097.0 655360.0 159737.1 7569408.0 7202856.3 43280.0 25855.8 5271 268.378 4969 9882.636 10151.014
> 81920.0 81920.0 0.0 4097.0 655360.0 159737.1 7569408.0 7202856.3 43280.0 25855.8 5271 268.378 4970 9882.782 10151.159
> 81920.0 81920.0 0.0 4097.0 655360.0 159958.2 7569408.0 7202856.3 43280.0 25855.8 5271 268.378 4970 9882.782 10151.159
> 81920.0 81920.0 0.0 4097.0 655360.0 159985.0 7569408.0 7202856.3 43280.0 25855.8 5271 268.378 4970 9882.782 10151.159
> 81920.0 81920.0 0.0 4097.0 655360.0 160249.0 7569408.0 7202856.3 43280.0 25855.8 5271 268.378 4971 9882.974 10151.352
> 81920.0 81920.0 0.0 4097.0 655360.0 160249.0 7569408.0 7202856.3 43280.0 25855.8 5271 268.378 4972 9883.121 10151.499
> 81920.0 81920.0 0.0 4097.0 655360.0 160361.3 7569408.0 7202856.3 43280.0 25855.8 5271 268.378 4972 9883.121 10151.499
> 81920.0 81920.0 0.0 4097.0 655360.0 160638.2 7569408.0 7202856.3 43280.0 25855.8 5271 268.378 4972 9883.121 10151.499
> 81920.0 81920.0 0.0 4097.0 655360.0 160894.9 7569408.0 7202856.3 43280.0 25855.8 5271 268.378 4973 9883.368 10151.746
> 81920.0 81920.0 0.0 4097.0 655360.0 160920.7 7569408.0 7202856.3 43280.0 25855.8 5271 268.378 4974 9883.519 10151.897
> 81920.0 81920.0 0.0 4097.0 655360.0 160920.7 7569408.0 7202856.3 43280.0 25855.8 5271 268.378 4974 9883.519 10151.897
> 81920.0 81920.0 0.0 4097.0 655360.0 161139.5 7569408.0 7202856.3 43280.0 25855.8 5271 268.378 4975 9883.519 10151.897
> 81920.0 81920.0 0.0 4097.0 655360.0 161168.4 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4975 9883.735 10152.113
> 81920.0 81920.0 0.0 4097.0 655360.0 161427.2 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4976 9883.884 10152.262
> 81920.0 81920.0 0.0 4097.0 655360.0 161446.2 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4976 9883.884 10152.262
> 81920.0 81920.0 0.0 4097.0 655360.0 161490.9 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4976 9883.884 10152.262
> 81920.0 81920.0 0.0 4097.0 655360.0 161673.2 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4977 9884.066 10152.444
> 81920.0 81920.0 0.0 4097.0 655360.0 162002.3 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4978 9884.217 10152.595
> 81920.0 81920.0 0.0 4097.0 655360.0 162023.6 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4978 9884.217 10152.595
> 81920.0 81920.0 0.0 4097.0 655360.0 162086.0 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4978 9884.217 10152.595
> 81920.0 81920.0 0.0 4097.0 655360.0 162271.9 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4979 9884.405 10152.782
> 81920.0 81920.0 0.0 4097.0 655360.0 162537.0 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4980 9884.557 10152.935
> 81920.0 81920.0 0.0 4097.0 655360.0 162696.2 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4980 9884.557 10152.935
> 81920.0 81920.0 0.0 4097.0 655360.0 162696.2 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4980 9884.557 10152.935
> 81920.0 81920.0 0.0 4097.0 655360.0 162740.8 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4981 9884.748 10153.125
> 81920.0 81920.0 0.0 4097.0 655360.0 163009.5 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4982 9884.901 10153.279
> 81920.0 81920.0 0.0 4097.0 655360.0 163285.1 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4982 9884.901 10153.279
> 81920.0 81920.0 0.0 4097.0 655360.0 163285.1 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4982 9884.901 10153.279
> 81920.0 81920.0 0.0 4097.0 655360.0 163397.4 7569408.0 7202856.1 43280.0 25855.8 5271 268.378 4983 9885.093 10153.471
> 81920.0 81920.0 0.0 4097.0 655360.0 163590.2 7569408.0 7202855.9 43280.0 25855.8 5271 268.378 4984 9885.251 10153.628
> 81920.0 81920.0 0.0 4097.0 655360.0 165038.0 7569408.0 7202855.9 43280.0 25855.8 5271 268.378 4984 9885.251 10153.628
> 81920.0 81920.0 0.0 4097.0 655360.0 165142.5 7569408.0 7202855.9 43280.0 25855.8 5271 268.378 4984 9885.251 10153.628
> 81920.0 81920.0 0.0 4097.0 655360.0 165142.5 7569408.0 7202855.9 43280.0 25855.8 5271 268.378 4985 9885.435 10153.813
> 81920.0 81920.0 0.0 4097.0 655360.0 165391.7 7569408.0 7202855.8 43280.0 25855.8 5271 268.378 4986 9885.588 10153.966
> 81920.0 81920.0 0.0 4097.0 655360.0 165401.9 7569408.0 7202855.8 43280.0 25855.8 5271 268.378 4986 9885.588 10153.966
> 81920.0 81920.0 0.0 4097.0 655360.0 165665.9 7569408.0 7202855.8 43280.0 25855.8 5271 268.378 4986 9885.588 10153.966
> 81920.0 81920.0 0.0 4097.0 655360.0 165667.9 7569408.0 7202855.8 43280.0 25855.8 5271 268.378 4987 9885.848 10154.226
> 81920.0 81920.0 0.0 4097.0 655360.0 165712.6 7569408.0 7202855.8 43280.0 25855.8 5271 268.378 4988 9886.002 10154.379
> 81920.0 81920.0 0.0 4097.0 655360.0 165971.0 7569408.0 7202855.8 43280.0 25855.8 5271 268.378 4988 9886.002 10154.379
> 81920.0 81920.0 0.0 4097.0 655360.0 166254.9 7569408.0 7202855.8 43280.0 25855.8 5271 268.378 4988 9886.002 10154.379
> 81920.0 81920.0 0.0 4097.0 655360.0 166257.1 7569408.0 7202855.8 43280.0 25855.8 5271 268.378 4989 9886.256 10154.633
> 81920.0 81920.0 0.0 4097.0 655360.0 166257.1 7569408.0 7202855.7 43280.0 25855.8 5271 268.378 4989 9886.256 10154.633
> 81920.0 81920.0 0.0 4097.0 655360.0 166496.5 7569408.0 7202855.7 43280.0 25855.8 5271 268.378 4990 9886.412 10154.789
> 81920.0 81920.0 0.0 4097.0 655360.0 166843.1 7569408.0 7202855.7 43280.0 25855.8 5271 268.378 4990 9886.412 10154.789
> 81920.0 81920.0 0.0 4097.0 655360.0 166852.0 7569408.0 7202855.7 43280.0 25855.8 5271 268.378 4991 9886.605 10154.983
> 81920.0 81920.0 0.0 4097.0 655360.0 166852.0 7569408.0 7202855.6 43280.0 25855.8 5271 268.378 4991 9886.605 10154.983
> 81920.0 81920.0 0.0 4097.0 655360.0 166908.3 7569408.0 7202855.6 43280.0 25855.8 5271 268.378 4992 9886.762 10155.140
> 81920.0 81920.0 0.0 4097.0 655360.0 167122.1 7569408.0 7202855.6 43280.0 25855.8 5271 268.378 4992 9886.762 10155.140
> 81920.0 81920.0 0.0 4097.0 655360.0 167375.4 7569408.0 7202855.6 43280.0 25855.8 5271 268.378 4993 9887.023 10155.401
> 81920.0 81920.0 0.0 4097.0 655360.0 167375.4 7569408.0 7202855.6 43280.0 25855.8 5271 268.378 4993 9887.023 10155.401
> 81920.0 81920.0 0.0 4097.0 655360.0 167377.6 7569408.0 7202855.6 43280.0 25855.8 5271 268.378 4994 9887.179 10155.557
> 81920.0 81920.0 0.0 4097.0 655360.0 167683.0 7569408.0 7202855.6 43280.0 25855.8 5271 268.378 4994 9887.179 10155.557
> 81920.0 81920.0 0.0 4097.0 655360.0 167948.1 7569408.0 7202855.6 43280.0 25855.8 5271 268.378 4994 9887.179 10155.557
> 81920.0 81920.0 0.0 4097.0 655360.0 167968.7 7569408.0 7202855.6 43280.0 25855.8 5271 268.378 4995 9887.376 10155.754
> 81920.0 81920.0 0.0 4097.0 655360.0 168034.2 7569408.0 7202855.6 43280.0 25855.8 5271 268.378 4996 9887.534 10155.911
> 81920.0 81920.0 0.0 4097.0 655360.0 168094.6 7569408.0 7202855.6 43280.0 25855.8 5271 268.378 4996 9887.534 10155.911
> 81920.0 81920.0 0.0 4097.0 655360.0 168270.8 7569408.0 7202855.6 43280.0 25855.8 5271 268.378 4996 9887.534 10155.911
> 81920.0 81920.0 0.0 4097.0 655360.0 168561.5 7569408.0 7202855.6 43280.0 25855.8 5271 268.378 4997 9887.808 10156.185
> 81920.0 81920.0 0.0 4097.0 655360.0 168627.3 7569408.0 7202855.5 43280.0 25855.8 5271 268.378 4998 9887.968 10156.345
> 81920.0 81920.0 0.0 4097.0 655360.0 168641.1 7569408.0 7202855.5 43280.0 25855.8 5271 268.378 4998 9887.968 10156.345
> 81920.0 81920.0 0.0 4097.0 655360.0 168859.9 7569408.0 7202855.5 43280.0 25855.8 5271 268.378 4998 9887.968 10156.345
> 81920.0 81920.0 0.0 4097.0 655360.0 169143.9 7569408.0 7202855.5 43280.0 25855.8 5271 268.378 4999 9888.170 10156.547
> 81920.0 81920.0 0.0 4097.0 655360.0 169152.9 7569408.0 7202855.4 43280.0 25855.8 5271 268.378 5000 9888.330 10156.708
> 81920.0 81920.0 0.0 4097.0 655360.0 169152.9 7569408.0 7202855.4 43280.0 25855.8 5271 268.378 5000 9888.330 10156.708
> 81920.0 81920.0 0.0 4097.0 655360.0 169373.9 7569408.0 7202855.4 43280.0 25855.8 5271 268.378 5000 9888.330 10156.708
> 81920.0 81920.0 0.0 4097.0 655360.0 169639.0 7569408.0 7202855.4 43280.0 25855.8 5271 268.378 5001 9888.591 10156.968
> 81920.0 81920.0 0.0 4097.0 655360.0 169744.2 7569408.0 7202855.4 43280.0 25855.8 5271 268.378 5002 9888.754 10157.131
> 81920.0 81920.0 0.0 4097.0 655360.0 169822.8 7569408.0 7202855.4 43280.0 25855.8 5271 268.378 5002 9888.754 10157.131
> 81920.0 81920.0 0.0 4097.0 655360.0 169888.3 7569408.0 7202855.4 43280.0 25855.8 5271 268.378 5002 9888.754 10157.131
> 81920.0 81920.0 0.0 4097.0 655360.0 170125.7 7569408.0 7202855.4 43280.0 25855.8 5271 268.378 5003 9888.946 10157.323
> 81920.0 81920.0 0.0 4097.0 655360.0 170399.8 7569408.0 7202855.3 43280.0 25855.8 5271 268.378 5004 9889.106 10157.483
> 81920.0 81920.0 0.0 4097.0 655360.0 170399.8 7569408.0 7202855.3 43280.0 25855.8 5271 268.378 5004 9889.106 10157.483
> 81920.0 81920.0 0.0 4097.0 655360.0 170420.7 7569408.0 7202855.3 43280.0 25855.8 5271 268.378 5004 9889.106 10157.483
> 81920.0 81920.0 0.0 4097.0 655360.0 170651.4 7569408.0 7202855.3 43280.0 25855.8 5271 268.378 5005 9889.301 10157.679
> 81920.0 81920.0 0.0 4097.0 655360.0 170916.8 7569408.0 7202855.2 43280.0 25855.8 5271 268.378 5006 9889.464 10157.842
> 81920.0 81920.0 0.0 4097.0 655360.0 170923.6 7569408.0 7202855.2 43280.0 25855.8 5271 268.378 5006 9889.464 10157.842
> 81920.0 81920.0 0.0 4097.0 655360.0 170925.6 7569408.0 7202855.2 43280.0 25855.8 5271 268.378 5006 9889.464 10157.842
> 81920.0 81920.0 0.0 4097.0 655360.0 171009.8 7569408.0 7202855.2 43280.0 25855.8 5271 268.378 5007 9889.660 10158.037
> 81920.0 81920.0 0.0 4097.0 655360.0 171249.8 7569408.0 7202855.2 43280.0 25855.8 5271 268.378 5008 9889.823 10158.201
> 81920.0 81920.0 0.0 4097.0 655360.0 171518.8 7569408.0 7202855.2 43280.0 25855.8 5271 268.378 5008 9889.823 10158.201
> 81920.0 81920.0 0.0 4097.0 655360.0 171530.6 7569408.0 7202855.2 43280.0 25855.8 5271 268.378 5008 9889.823 10158.201
> 81920.0 81920.0 0.0 4097.0 655360.0 171549.2 7569408.0 7202855.2 43280.0 25855.8 5271 268.378 5009 9890.024 10158.402
> 81920.0 81920.0 0.0 4097.0 655360.0 173035.0 7569408.0 7202850.4 43280.0 25855.8 5271 268.378 5010 9890.190 10158.568
> 81920.0 81920.0 0.0 4097.0 655360.0 173367.8 7569408.0 7202850.4 43280.0 25855.8 5271 268.378 5010 9890.190 10158.568
> 81920.0 81920.0 0.0 4097.0 655360.0 173376.8 7569408.0 7202850.4 43280.0 25855.8 5271 268.378 5010 9890.190 10158.568
> 81920.0 81920.0 0.0 4097.0 655360.0 173395.5 7569408.0 7202850.4 43280.0 25855.8 5271 268.378 5011 9890.456 10158.834
> 81920.0 81920.0 0.0 4097.0 655360.0 173451.8 7569408.0 7202850.3 43280.0 25855.8 5271 268.378 5011 9890.456 10158.834
> 81920.0 81920.0 0.0 4097.0 655360.0 173933.5 7569408.0 7202850.3 43280.0 25855.8 5271 268.378 5012 9890.622 10158.999
> 81920.0 81920.0 0.0 4097.0 655360.0 173969.7 7569408.0 7202850.3 43280.0 25855.8 5271 268.378 5012 9890.622 10158.999
> 81920.0 81920.0 0.0 4097.0 655360.0 173988.3 7569408.0 7202850.3 43280.0 25855.8 5271 268.378 5013 9890.820 10159.197
> 81920.0 81920.0 0.0 4097.0 655360.0 174053.8 7569408.0 7202850.2 43280.0 25855.8 5271 268.378 5013 9890.820 10159.197
> 81920.0 81920.0 0.0 4097.0 655360.0 174286.1 7569408.0 7202850.2 43280.0 25855.8 5271 268.378 5014 9890.986 10159.363
> 81920.0 81920.0 0.0 4097.0 655360.0 174551.0 7569408.0 7202850.2 43280.0 25855.8 5271 268.378 5014 9890.986 10159.363
> 81920.0 81920.0 0.0 4097.0 655360.0 174572.0 7569408.0 7202850.2 43280.0 25855.8 5271 268.378 5015 9890.986 10159.363
> 81920.0 81920.0 0.0 4097.0 655360.0 174592.9 7569408.0 7202850.1 43280.0 25855.8 5271 268.378 5015 9891.185 10159.562
> 81920.0 81920.0 0.0 4097.0 655360.0 174637.4 7569408.0 7202850.1 43280.0 25855.8 5271 268.378 5016 9891.350 10159.728
> 81920.0 81920.0 0.0 4097.0 655360.0 175049.8 7569408.0 7202850.1 43280.0 25855.8 5271 268.378 5016 9891.350 10159.728
> 81920.0 81920.0 0.0 4097.0 655360.0 175130.7 7569408.0 7202850.1 43280.0 25855.8 5271 268.378 5016 9891.350 10159.728
> 81920.0 81920.0 0.0 4097.0 655360.0 175168.4 7569408.0 7202850.1 43280.0 25855.8 5271 268.378 5017 9891.554 10159.932
> 81920.0 81920.0 0.0 4097.0 655360.0 175182.2 7569408.0 7202850.1 43280.0 25855.8 5271 268.378 5018 9891.721 10160.098
> 81920.0 81920.0 0.0 4097.0 655360.0 175402.9 7569408.0 7202850.1 43280.0 25855.8 5271 268.378 5018 9891.721 10160.098
> 81920.0 81920.0 0.0 4097.0 655360.0 175723.5 7569408.0 7202850.1 43280.0 25855.8 5271 268.378 5018 9891.721 10160.098
> 81920.0 81920.0 0.0 4097.0 655360.0 175742.2 7569408.0 7202850.1 43280.0 25855.8 5271 268.378 5019 9891.921 10160.298
> 81920.0 81920.0 0.0 4097.0 655360.0 175778.9 7569408.0 7202850.0 43280.0 25855.8 5271 268.378 5020 9892.089 10160.466
> 81920.0 81920.0 0.0 4097.0 655360.0 175823.6 7569408.0 7202850.0 43280.0 25855.8 5271 268.378 5020 9892.089 10160.466
> 81920.0 81920.0 0.0 4097.0 655360.0 176315.6 7569408.0 7202850.0 43280.0 25855.8 5271 268.378 5020 9892.089 10160.466
> 81920.0 81920.0 0.0 4097.0 655360.0 176351.6 7569408.0 7202850.0 43280.0 25855.8 5271 268.378 5021 9892.291 10160.669
> 81920.0 81920.0 0.0 4097.0 655360.0 176351.6 7569408.0 7202849.9 43280.0 25855.8 5271 268.378 5022 9892.460 10160.838
> 81920.0 81920.0 0.0 4097.0 655360.0 176370.6 7569408.0 7202849.9 43280.0 25855.8 5271 268.378 5022 9892.460 10160.838
> 81920.0 81920.0 0.0 4097.0 655360.0 176846.1 7569408.0 7202849.9 43280.0 25855.8 5271 268.378 5022 9892.460 10160.838
> 81920.0 81920.0 0.0 4097.0 655360.0 176865.2 7569408.0 7202849.9 43280.0 25855.8 5271 268.378 5023 9892.665 10161.043
> 81920.0 81920.0 0.0 4097.0 655360.0 176865.2 7569408.0 7202849.9 43280.0 25855.8 5271 268.378 5024 9892.834 10161.212
> 81920.0 81920.0 0.0 4097.0 655360.0 176942.8 7569408.0 7202849.9 43280.0 25855.8 5271 268.378 5024 9892.834 10161.212
> 81920.0 81920.0 0.0 4097.0 655360.0 177025.1 7569408.0 7202849.9 43280.0 25855.8 5271 268.378 5024 9892.834 10161.212
> 81920.0 81920.0 0.0 4097.0 655360.0 177437.4 7569408.0 7202849.9 43280.0 25855.8 5271 268.378 5025 9893.039 10161.416
> 81920.0 81920.0 0.0 4097.0 655360.0 177468.4 7569408.0 7202849.9 43280.0 25855.8 5271 268.378 5026 9893.213 10161.590
> 81920.0 81920.0 0.0 4097.0 655360.0 177472.6 7569408.0 7202849.9 43280.0 25855.8 5271 268.378 5026 9893.213 10161.590
> 81920.0 81920.0 0.0 4097.0 655360.0 177556.8 7569408.0 7202849.9 43280.0 25855.8 5271 268.378 5026 9893.213 10161.590
> 81920.0 81920.0 0.0 4097.0 655360.0 178032.6 7569408.0 7202849.9 43280.0 25855.8 5271 268.378 5027 9893.454 10161.832
> 81920.0 81920.0 0.0 4097.0 655360.0 178059.4 7569408.0 7202849.8 43280.0 25855.8 5271 268.378 5028 9893.624 10162.001
> 81920.0 81920.0 0.0 4097.0 655360.0 178059.4 7569408.0 7202849.8 43280.0 25855.8 5271 268.378 5028 9893.624 10162.001
> 81920.0 81920.0 0.0 4097.0 655360.0 178059.4 7569408.0 7202849.8 43280.0 25855.8 5271 268.378 5028 9893.624 10162.001
> 81920.0 81920.0 0.0 4097.0 655360.0 178300.8 7569408.0 7202849.8 43280.0 25855.8 5271 268.378 5029 9893.901 10162.279
> 81920.0 81920.0 0.0 4097.0 655360.0 178630.4 7569408.0 7202849.7 43280.0 25855.8 5271 268.378 5030 9894.075 10162.453
> 81920.0 81920.0 0.0 4097.0 655360.0 178638.6 7569408.0 7202849.7 43280.0 25855.8 5271 268.378 5030 9894.075 10162.453
> 81920.0 81920.0 0.0 4097.0 655360.0 178638.6 7569408.0 7202849.7 43280.0 25855.8 5271 268.378 5030 9894.075 10162.453
> 81920.0 81920.0 0.0 4097.0 655360.0 178734.5 7569408.0 7202849.7 43280.0 25855.8 5271 268.378 5031 9894.294 10162.672
> 81920.0 81920.0 0.0 4097.0 655360.0 179272.2 7569408.0 7202849.7 43280.0 25855.8 5271 268.378 5032 9894.468 10162.845
> 81920.0 81920.0 0.0 4097.0 655360.0 179308.5 7569408.0 7202849.7 43280.0 25855.8 5271 268.378 5032 9894.468 10162.845
> 81920.0 81920.0 0.0 4097.0 655360.0 179308.5 7569408.0 7202849.7 43280.0 25855.8 5271 268.378 5032 9894.468 10162.845
> 81920.0 81920.0 0.0 4097.0 655360.0 179308.5 7569408.0 7202849.7 43280.0 25855.8 5271 268.378 5033 9894.681 10163.058
> 81920.0 81920.0 0.0 4097.0 655360.0 179784.0 7569408.0 7202849.6 43280.0 25855.8 5271 268.378 5033 9894.681 10163.058
> 81920.0 81920.0 0.0 4097.0 655360.0 179804.6 7569408.0 7202849.6 43280.0 25855.8 5271 268.378 5034 9894.854 10163.231
> 81920.0 81920.0 0.0 4097.0 655360.0 181033.4 7569408.0 7202849.6 43280.0 25855.8 5271 268.378 5034 9894.854 10163.231
> 81920.0 81920.0 0.0 4097.0 655360.0 181033.4 7569408.0 7202849.6 43280.0 25855.8 5271 268.378 5035 9895.135 10163.513
> 81920.0 81920.0 0.0 4097.0 655360.0 181052.0 7569408.0 7202849.5 43280.0 25855.8 5271 268.378 5035 9895.135 10163.513
> 81920.0 81920.0 0.0 4097.0 655360.0 181509.0 7569408.0 7202849.5 43280.0 25855.8 5271 268.378 5036 9895.311 10163.689
> 81920.0 81920.0 0.0 4097.0 655360.0 181608.5 7569408.0 7202849.5 43280.0 25855.8 5271 268.378 5036 9895.311 10163.689
> 81920.0 81920.0 0.0 4097.0 655360.0 181610.7 7569408.0 7202849.5 43280.0 25855.8 5271 268.378 5037 9895.311 10163.689
> 81920.0 81920.0 0.0 4097.0 655360.0 181640.3 7569408.0 7202849.5 43280.0 25855.8 5271 268.378 5037 9895.539 10163.917
> 81920.0 81920.0 0.0 4097.0 655360.0 182116.0 7569408.0 7202849.5 43280.0 25855.8 5271 268.378 5038 9895.716 10164.093
> 81920.0 81920.0 0.0 4097.0 655360.0 182207.3 7569408.0 7202849.5 43280.0 25855.8 5271 268.378 5038 9895.716 10164.093
> 81920.0 81920.0 0.0 4097.0 655360.0 182281.0 7569408.0 7202849.5 43280.0 25855.8 5271 268.378 5038 9895.716 10164.093
> 81920.0 81920.0 0.0 4097.0 655360.0 182281.0 7569408.0 7202849.5 43280.0 25855.8 5271 268.378 5039 9896.000 10164.378
> 81920.0 81920.0 0.0 4097.0 655360.0 182285.1 7569408.0 7202849.5 43280.0 25855.8 5271 268.378 5040 9896.180 10164.558
> 81920.0 81920.0 0.0 4097.0 655360.0 182844.7 7569408.0 7202849.5 43280.0 25855.8 5271 268.378 5040 9896.180 10164.558
> 81920.0 81920.0 0.0 4097.0 655360.0 182872.0 7569408.0 7202849.5 43280.0 25855.8 5271 268.378 5040 9896.180 10164.558
> 81920.0 81920.0 0.0 4097.0 655360.0 182872.0 7569408.0 7202849.5 43280.0 25855.8 5271 268.378 5041 9896.463 10164.841
> 81920.0 81920.0 0.0 4097.0 655360.0 182872.0 7569408.0 7202849.3 43280.0 25855.8 5271 268.378 5042 9896.641 10165.019
> 81920.0 81920.0 0.0 4097.0 655360.0 183413.3 7569408.0 7202849.3 43280.0 25855.8 5271 268.378 5042 9896.641 10165.019
> 81920.0 81920.0 0.0 4097.0 655360.0 183443.4 7569408.0 7202849.3 43280.0 25855.8 5271 268.378 5042 9896.641 10165.019
> 81920.0 81920.0 0.0 4097.0 655360.0 183464.8 7569408.0 7202849.3 43280.0 25855.8 5271 268.378 5043 9896.857 10165.235
> 81920.0 81920.0 0.0 4097.0 655360.0 183476.6 7569408.0 7202849.2 43280.0 25855.8 5271 268.378 5044 9897.033 10165.411
> 81920.0 81920.0 0.0 4097.0 655360.0 183733.3 7569408.0 7202849.2 43280.0 25855.8 5271 268.378 5044 9897.033 10165.411
> 81920.0 81920.0 0.0 4097.0 655360.0 184045.6 7569408.0 7202849.2 43280.0 25855.8 5271 268.378 5044 9897.033 10165.411
> 81920.0 81920.0 0.0 4097.0 655360.0 184055.9 7569408.0 7202849.2 43280.0 25855.8 5271 268.378 5045 9897.314 10165.691
> 81920.0 81920.0 0.0 4097.0 655360.0 184057.9 7569408.0 7202849.2 43280.0 25855.8 5271 268.378 5046 9897.492 10165.869
> 81920.0 81920.0 0.0 4097.0 655360.0 184057.9 7569408.0 7202849.2 43280.0 25855.8 5271 268.378 5046 9897.492 10165.869
> 81920.0 81920.0 0.0 4097.0 655360.0 184618.0 7569408.0 7202849.2 43280.0 25855.8 5271 268.378 5046 9897.492 10165.869
> 81920.0 81920.0 0.0 4097.0 655360.0 184644.8 7569408.0 7202849.2 43280.0 25855.8 5271 268.378 5047 9897.706 10166.084
> 81920.0 81920.0 0.0 4097.0 655360.0 184647.0 7569408.0 7202849.1 43280.0 25855.8 5271 268.378 5048 9897.886 10166.264
> 81920.0 81920.0 0.0 4097.0 655360.0 184647.0 7569408.0 7202849.1 43280.0 25855.8 5271 268.378 5048 9897.886 10166.264
> 81920.0 81920.0 0.0 4097.0 655360.0 185105.9 7569408.0 7202849.1 43280.0 25855.8 5271 268.378 5048 9897.886 10166.264
> 81920.0 81920.0 0.0 4097.0 655360.0 185231.6 7569408.0 7202849.1 43280.0 25855.8 5271 268.378 5049 9898.095 10166.472
> 81920.0 81920.0 0.0 4097.0 655360.0 185239.8 7569408.0 7202849.1 43280.0 25855.8 5271 268.378 5050 9898.276 10166.654
> 81920.0 81920.0 0.0 4097.0 655360.0 185251.5 7569408.0 7202849.1 43280.0 25855.8 5271 268.378 5050 9898.276 10166.654
> 81920.0 81920.0 0.0 4097.0 655360.0 194817.6 7569408.0 7202849.1 43280.0 25855.8 5271 268.378 5050 9898.276 10166.654
> 81920.0 81920.0 0.0 4097.0 655360.0 195326.2 7569408.0 7202849.1 43280.0 25855.8 5271 268.378 5051 9898.570 10166.948
> 81920.0 81920.0 0.0 4097.0 655360.0 195333.0 7569408.0 7202849.0 43280.0 25855.8 5271 268.378 5052 9898.752 10167.130
> 81920.0 81920.0 0.0 4097.0 655360.0 195341.3 7569408.0 7202849.0 43280.0 25855.8 5271 268.378 5052 9898.752 10167.130
> 81920.0 81920.0 0.0 4097.0 655360.0 195341.3 7569408.0 7202849.0 43280.0 25855.8 5271 268.378 5052 9898.752 10167.130
> 81920.0 81920.0 0.0 4097.0 655360.0 196751.1 7569408.0 7202849.0 43280.0 25855.8 5271 268.378 5053 9899.045 10167.423
> 81920.0 81920.0 0.0 4097.0 655360.0 196934.4 7569408.0 7202848.8 43280.0 25855.8 5271 268.378 5053 9899.045 10167.423
> 81920.0 81920.0 0.0 4097.0 655360.0 196944.7 7569408.0 7202848.8 43280.0 25855.8 5271 268.378 5054 9899.230 10167.608
> 81920.0 81920.0 0.0 4097.0 655360.0 197021.9 7569408.0 7202848.8 43280.0 25855.8 5271 268.378 5054 9899.230 10167.608
> 81920.0 81920.0 0.0 4097.0 655360.0 197037.8 7569408.0 7202848.8 43280.0 25855.8 5271 268.378 5055 9899.451 10167.829
> 81920.0 81920.0 0.0 4097.0 655360.0 197541.2 7569408.0 7202848.7 43280.0 25855.8 5271 268.378 5055 9899.451 10167.829
> 81920.0 81920.0 0.0 4097.0 655360.0 197541.2 7569408.0 7202848.7 43280.0 25855.8 5271 268.378 5056 9899.638 10168.015
> 81920.0 81920.0 0.0 4097.0 655360.0 197614.9 7569408.0 7202848.7 43280.0 25855.8 5271 268.378 5056 9899.638 10168.015
> 81920.0 81920.0 0.0 4097.0 655360.0 197617.0 7569408.0 7202848.7 43280.0 25855.8 5271 268.378 5057 9899.858 10168.235
> 81920.0 81920.0 0.0 4097.0 655360.0 198123.4 7569408.0 7202848.7 43280.0 25855.8 5271 268.378 5057 9899.858 10168.235
> 81920.0 81920.0 0.0 4097.0 655360.0 198130.2 7569408.0 7202848.7 43280.0 25855.8 5271 268.378 5058 9900.043 10168.420
> 81920.0 81920.0 0.0 4097.0 655360.0 198140.7 7569408.0 7202848.7 43280.0 25855.8 5271 268.378 5058 9900.043 10168.420
> 81920.0 81920.0 0.0 4097.0 655360.0 199340.2 7569408.0 7202848.7 43280.0 25855.8 5271 268.378 5059 9900.340 10168.718
> 81920.0 81920.0 0.0 4097.0 655360.0 199580.3 7569408.0 7202848.6 43280.0 25855.8 5271 268.378 5059 9900.340 10168.718
> 81920.0 81920.0 0.0 4097.0 655360.0 199843.5 7569408.0 7202848.6 43280.0 25855.8 5271 268.378 5060 9900.528 10168.905
> 81920.0 81920.0 0.0 4097.0 655360.0 199917.3 7569408.0 7202848.6 43280.0 25855.8 5271 268.378 5060 9900.528 10168.905
> 81920.0 81920.0 0.0 4097.0 655360.0 200007.7 7569408.0 7202848.6 43280.0 25855.8 5271 268.378 5061 9900.528 10168.905
> 81920.0 81920.0 0.0 4097.0 655360.0 200009.8 7569408.0 7202848.6 43280.0 25855.8 5271 268.378 5061 9900.818 10169.196
> 81920.0 81920.0 0.0 4097.0 655360.0 200466.7 7569408.0 7202848.6 43280.0 25855.8 5271 268.378 5062 9901.005 10169.383
> 81920.0 81920.0 0.0 4097.0 655360.0 200513.2 7569408.0 7202848.6 43280.0 25855.8 5271 268.378 5062 9901.005 10169.383
> 81920.0 81920.0 0.0 4097.0 655360.0 200521.4 7569408.0 7202848.6 43280.0 25855.8 5271 268.378 5062 9901.005 10169.383
> 81920.0 81920.0 0.0 4097.0 655360.0 200521.4 7569408.0 7202848.5 43280.0 25855.8 5271 268.378 5063 9901.302 10169.680
> 81920.0 81920.0 0.0 4097.0 655360.0 200759.4 7569408.0 7202848.5 43280.0 25855.8 5271 268.378 5064 9901.490 10169.868
> 81920.0 81920.0 0.0 4097.0 655360.0 201018.1 7569408.0 7202848.5 43280.0 25855.8 5271 268.378 5064 9901.490 10169.868
> 81920.0 81920.0 0.0 4097.0 655360.0 201045.0 7569408.0 7202848.5 43280.0 25855.8 5271 268.378 5064 9901.490 10169.868
> 81920.0 81920.0 0.0 4097.0 655360.0 201112.5 7569408.0 7202848.5 43280.0 25855.8 5271 268.378 5065 9901.710 10170.087
> 81920.0 81920.0 0.0 4097.0 655360.0 201112.5 7569408.0 7202848.5 43280.0 25855.8 5271 268.378 5066 9901.900 10170.277
> 81920.0 81920.0 0.0 4097.0 655360.0 201590.6 7569408.0 7202848.5 43280.0 25855.8 5271 268.378 5066 9901.900 10170.277
> 81920.0 81920.0 0.0 4097.0 655360.0 201613.2 7569408.0 7202848.5 43280.0 25855.8 5271 268.378 5066 9901.900 10170.277
> 81920.0 81920.0 0.0 4097.0 655360.0 201640.1 7569408.0 7202848.5 43280.0 25855.8 5271 268.378 5067 9902.200 10170.578
> 81920.0 81920.0 0.0 4097.0 655360.0 201717.4 7569408.0 7202848.4 43280.0 25855.8 5271 268.378 5068 9902.390 10170.768
> 81920.0 81920.0 0.0 4097.0 655360.0 202023.1 7569408.0 7202848.4 43280.0 25855.8 5271 268.378 5068 9902.390 10170.768
> 81920.0 81920.0 0.0 4097.0 655360.0 202267.7 7569408.0 7202848.4 43280.0 25855.8 5271 268.378 5068 9902.390 10170.768
> 81920.0 81920.0 0.0 4097.0 655360.0 202278.1 7569408.0 7202848.4 43280.0 25855.8 5271 268.378 5069 9902.693 10171.071
> 81920.0 81920.0 0.0 4097.0 655360.0 202362.5 7569408.0 7202848.4 43280.0 25855.8 5271 268.378 5070 9902.883 10171.261
> 81920.0 81920.0 0.0 4097.0 655360.0 202600.7 7569408.0 7202848.4 43280.0 25855.8 5271 268.378 5070 9902.883 10171.261
> 81920.0 81920.0 0.0 4097.0 655360.0 202854.2 7569408.0 7202848.4 43280.0 25855.8 5271 268.378 5070 9902.883 10171.261
> 81920.0 81920.0 0.0 4097.0 655360.0 202926.5 7569408.0 7202848.3 43280.0 25855.8 5271 268.378 5071 9903.113 10171.491
> 81920.0 81920.0 0.0 4097.0 655360.0 202934.8 7569408.0 7202848.2 43280.0 25855.8 5271 268.378 5072 9903.303 10171.681
> 81920.0 81920.0 0.0 4097.0 655360.0 202955.5 7569408.0 7202848.2 43280.0 25855.8 5271 268.378 5072 9903.303 10171.681
> 81920.0 81920.0 0.0 4097.0 655360.0 203291.3 7569408.0 7202848.2 43280.0 25855.8 5271 268.378 5072 9903.303 10171.681
> 81920.0 81920.0 0.0 4097.0 655360.0 203517.0 7569408.0 7202848.2 43280.0 25855.8 5271 268.378 5073 9903.607 10171.985
> 81920.0 81920.0 0.0 4097.0 655360.0 203539.0 7569408.0 7202848.0 43280.0 25855.8 5271 268.378 5074 9903.799 10172.177
> 81920.0 81920.0 0.0 4097.0 655360.0 203557.6 7569408.0 7202848.0 43280.0 25855.8 5271 268.378 5074 9903.799 10172.177
> 81920.0 81920.0 0.0 4097.0 655360.0 203795.7 7569408.0 7202848.0 43280.0 25855.8 5271 268.378 5074 9903.799 10172.177
> 81920.0 81920.0 0.0 4097.0 655360.0 204042.6 7569408.0 7202848.0 43280.0 25855.8 5271 268.378 5075 9904.025 10172.403
> 81920.0 81920.0 0.0 4097.0 655360.0 204042.6 7569408.0 7202848.0 43280.0 25855.8 5271 268.378 5076 9904.220 10172.598
> 81920.0 81920.0 0.0 4097.0 655360.0 204135.0 7569408.0 7202848.0 43280.0 25855.8 5271 268.378 5076 9904.220 10172.598
> 81920.0 81920.0 0.0 4097.0 655360.0 204137.0 7569408.0 7202848.0 43280.0 25855.8 5271 268.378 5076 9904.220 10172.598
> 81920.0 81920.0 0.0 4097.0 655360.0 204582.2 7569408.0 7202848.0 43280.0 25855.8 5271 268.378 5077 9904.451 10172.829
> 81920.0 81920.0 0.0 4097.0 655360.0 204633.6 7569408.0 7202847.9 43280.0 25855.8 5271 268.378 5078 9904.644 10173.022
> 81920.0 81920.0 0.0 4097.0 655360.0 204660.4 7569408.0 7202847.9 43280.0 25855.8 5271 268.378 5078 9904.644 10173.022
> 81920.0 81920.0 0.0 4097.0 655360.0 204741.6 7569408.0 7202847.9 43280.0 25855.8 5271 268.378 5078 9904.644 10173.022
> 81920.0 81920.0 0.0 4097.0 655360.0 205000.7 7569408.0 7202847.9 43280.0 25855.8 5271 268.378 5079 9904.952 10173.329
> 81920.0 81920.0 0.0 4097.0 655360.0 205226.3 7569408.0 7202847.8 43280.0 25855.8 5271 268.378 5080 9904.952 10173.329
> 81920.0 81920.0 0.0 4097.0 655360.0 205228.5 7569408.0 7202847.8 43280.0 25855.8 5271 268.378 5080 9905.144 10173.521
> 81920.0 81920.0 0.0 4097.0 655360.0 205267.2 7569408.0 7202847.8 43280.0 25855.8 5271 268.378 5080 9905.144 10173.521
> 81920.0 81920.0 0.0 4097.0 655360.0 205267.2 7569408.0 7202847.8 43280.0 25855.8 5271 268.378 5081 9905.369 10173.746
> 81920.0 81920.0 0.0 4097.0 655360.0 205698.6 7569408.0 7202847.8 43280.0 25855.8 5271 268.378 5081 9905.369 10173.746
> 81920.0 81920.0 0.0 4097.0 655360.0 205752.3 7569408.0 7202847.8 43280.0 25855.8 5271 268.378 5082 9905.563 10173.940
> 81920.0 81920.0 0.0 4097.0 655360.0 205844.6 7569408.0 7202847.8 43280.0 25855.8 5271 268.378 5082 9905.563 10173.940
> 81920.0 81920.0 0.0 4097.0 655360.0 205846.6 7569408.0 7202847.8 43280.0 25855.8 5271 268.378 5083 9905.797 10174.174
> 81920.0 81920.0 0.0 4097.0 655360.0 206115.6 7569408.0 7202847.8 43280.0 25855.8 5271 268.378 5083 9905.797 10174.174
> 81920.0 81920.0 0.0 4097.0 655360.0 206406.6 7569408.0 7202847.8 43280.0 25855.8 5271 268.378 5084 9905.992 10174.369
> 81920.0 81920.0 0.0 4097.0 655360.0 207640.6 7569408.0 7202847.8 43280.0 25855.8 5271 268.378 5084 9905.992 10174.369
> 81920.0 81920.0 0.0 4097.0 655360.0 207650.8 7569408.0 7202847.8 43280.0 25855.8 5271 268.378 5085 9906.220 10174.598
> 81920.0 81920.0 0.0 4097.0 655360.0 207908.0 7569408.0 7202847.7 43280.0 25855.8 5271 268.378 5085 9906.220 10174.598
> 81920.0 81920.0 0.0 4097.0 655360.0 208149.9 7569408.0 7202847.7 43280.0 25855.8 5271 268.378 5086 9906.417 10174.794
> 81920.0 81920.0 0.0 4097.0 655360.0 208201.3 7569408.0 7202847.7 43280.0 25855.8 5271 268.378 5086 9906.417 10174.794
> 81920.0 81920.0 0.0 4097.0 655360.0 208240.0 7569408.0 7202847.7 43280.0 25855.8 5271 268.378 5087 9906.417 10174.794
> 81920.0 81920.0 0.0 4097.0 655360.0 208240.0 7569408.0 7202847.6 43280.0 25855.8 5271 268.378 5087 9906.654 10175.031
> 81920.0 81920.0 0.0 4097.0 655360.0 208501.2 7569408.0 7202847.6 43280.0 25855.8 5271 268.378 5088 9906.853 10175.230
> 81920.0 81920.0 0.0 4097.0 655360.0 208726.8 7569408.0 7202847.6 43280.0 25855.8 5271 268.378 5088 9906.853 10175.230
> 81920.0 81920.0 0.0 4097.0 655360.0 208726.8 7569408.0 7202847.6 43280.0 25855.8 5271 268.378 5088 9906.853 10175.230
> 81920.0 81920.0 0.0 4097.0 655360.0 208821.1 7569408.0 7202847.6 43280.0 25855.8 5271 268.378 5089 9907.164 10175.542
> 81920.0 81920.0 0.0 4097.0 655360.0 209059.2 7569408.0 7202847.6 43280.0 25855.8 5271 268.378 5090 9907.365 10175.743
> 81920.0 81920.0 0.0 4097.0 655360.0 209398.1 7569408.0 7202847.6 43280.0 25855.8 5271 268.378 5090 9907.365 10175.743
> 81920.0 81920.0 0.0 4097.0 655360.0 209400.4 7569408.0 7202847.6 43280.0 25855.8 5271 268.378 5090 9907.365 10175.743
> 81920.0 81920.0 0.0 4097.0 655360.0 209408.6 7569408.0 7202847.6 43280.0 25855.8 5271 268.378 5091 9907.596 10175.973
> 81920.0 81920.0 0.0 4097.0 655360.0 209427.3 7569408.0 7202847.5 43280.0 25855.8 5271 268.378 5092 9907.793 10176.171
> 81920.0 81920.0 0.0 4097.0 655360.0 209684.5 7569408.0 7202847.5 43280.0 25855.8 5271 268.378 5092 9907.793 10176.171
> 81920.0 81920.0 0.0 4097.0 655360.0 209910.1 7569408.0 7202847.5 43280.0 25855.8 5271 268.378 5092 9907.793 10176.171
> 81920.0 81920.0 0.0 4097.0 655360.0 209932.1 7569408.0 7202847.5 43280.0 25855.8 5271 268.378 5093 9908.028 10176.405
> 81920.0 81920.0 0.0 4097.0 655360.0 209952.8 7569408.0 7202847.4 43280.0 25855.8 5271 268.378 5094 9908.225 10176.603
> 81920.0 81920.0 0.0 4097.0 655360.0 210190.9 7569408.0 7202847.4 43280.0 25855.8 5271 268.378 5094 9908.225 10176.603
> 81920.0 81920.0 0.0 4097.0 655360.0 210500.9 7569408.0 7202847.4 43280.0 25855.8 5271 268.378 5094 9908.225 10176.603
> 81920.0 81920.0 0.0 4097.0 655360.0 210500.9 7569408.0 7202847.4 43280.0 25855.8 5271 268.378 5095 9908.470 10176.848
> 81920.0 81920.0 0.0 4097.0 655360.0 210509.1 7569408.0 7202847.4 43280.0 25855.8 5271 268.378 5096 9908.668 10177.046
> 81920.0 81920.0 0.0 4097.0 655360.0 210785.8 7569408.0 7202847.4 43280.0 25855.8 5271 268.378 5096 9908.668 10177.046
> 81920.0 81920.0 0.0 4097.0 655360.0 210818.8 7569408.0 7202847.4 43280.0 25855.8 5271 268.378 5096 9908.668 10177.046
> 81920.0 81920.0 0.0 4097.0 655360.0 211109.8 7569408.0 7202847.4 43280.0 25855.8 5271 268.378 5097 9908.902 10177.279
> 81920.0 81920.0 0.0 4097.0 655360.0 211175.3 7569408.0 7202847.4 43280.0 25855.8 5271 268.378 5098 9909.103 10177.481
> 81920.0 81920.0 0.0 4097.0 655360.0 211267.7 7569408.0 7202847.4 43280.0 25855.8 5271 268.378 5098 9909.103 10177.481
> 81920.0 81920.0 0.0 4097.0 655360.0 211527.0 7569408.0 7202847.4 43280.0 25855.8 5271 268.378 5098 9909.103 10177.481
> 81920.0 81920.0 0.0 4097.0 655360.0 211754.7 7569408.0 7202847.4 43280.0 25855.8 5271 268.378 5099 9909.341 10177.718
> 81920.0 81920.0 0.0 4097.0 655360.0 211766.4 7569408.0 7202847.2 43280.0 25855.8 5271 268.378 5100 9909.542 10177.920
> 81920.0 81920.0 0.0 4097.0 655360.0 211840.2 7569408.0 7202847.2 43280.0 25855.8 5271 268.378 5100 9909.542 10177.920
> 81920.0 81920.0 0.0 4097.0 655360.0 212116.0 7569408.0 7202847.2 43280.0 25855.8 5271 268.378 5100 9909.542 10177.920
> 81920.0 81920.0 0.0 4097.0 655360.0 212122.8 7569408.0 7202847.2 43280.0 25855.8 5271 268.378 5101 9909.783 10178.161
> 81920.0 81920.0 0.0 4097.0 655360.0 212411.4 7569408.0 7202847.2 43280.0 25855.8 5271 268.378 5102 9909.783 10178.161
> 81920.0 81920.0 0.0 4097.0 655360.0 212424.6 7569408.0 7202847.2 43280.0 25855.8 5271 268.378 5102 9909.987 10178.365
> 81920.0 81920.0 0.0 4097.0 655360.0 212432.8 7569408.0 7202847.2 43280.0 25855.8 5271 268.378 5102 9909.987 10178.365
> 81920.0 81920.0 0.0 4097.0 655360.0 212720.2 7569408.0 7202847.2 43280.0 25855.8 5271 268.378 5103 9910.313 10178.690
> 81920.0 81920.0 0.0 4097.0 655360.0 212945.8 7569408.0 7202847.1 43280.0 25855.8 5271 268.378 5103 9910.313 10178.690
> 81920.0 81920.0 0.0 4097.0 655360.0 212945.8 7569408.0 7202847.1 43280.0 25855.8 5271 268.378 5104 9910.516 10178.893
> 81920.0 81920.0 0.0 4097.0 655360.0 212958.1 7569408.0 7202847.1 43280.0 25855.8 5271 268.378 5104 9910.516 10178.893
> 81920.0 81920.0 0.0 4097.0 655360.0 213301.3 7569408.0 7202847.1 43280.0 25855.8 5271 268.378 5105 9910.851 10179.229
> 81920.0 81920.0 0.0 4097.0 655360.0 213482.3 7569408.0 7202847.0 43280.0 25855.8 5271 268.378 5105 9910.851 10179.229
> 81920.0 81920.0 0.0 4097.0 655360.0 213527.0 7569408.0 7202847.0 43280.0 25855.8 5271 268.378 5106 9911.056 10179.433
> 81920.0 81920.0 0.0 4097.0 655360.0 213547.0 7569408.0 7202847.0 43280.0 25855.8 5271 268.378 5106 9911.056 10179.433
> 81920.0 81920.0 0.0 4097.0 655360.0 213612.5 7569408.0 7202847.0 43280.0 25855.8 5271 268.378 5107 9911.307 10179.684
> 81920.0 81920.0 0.0 4097.0 655360.0 213892.6 7569408.0 7202847.0 43280.0 25855.8 5271 268.378 5107 9911.307 10179.684
> 81920.0 81920.0 0.0 4097.0 655360.0 214133.8 7569408.0 7202847.0 43280.0 25855.8 5271 268.378 5108 9911.513 10179.891
> 81920.0 81920.0 0.0 4097.0 655360.0 214133.8 7569408.0 7202847.0 43280.0 25855.8 5271 268.378 5108 9911.513 10179.891
> 81920.0 81920.0 0.0 4097.0 655360.0 215407.3 7569408.0 7202847.0 43280.0 25855.8 5271 268.378 5108 9911.513 10179.891
> 81920.0 81920.0 0.0 4097.0 655360.0 215677.9 7569408.0 7202846.9 43280.0 25855.8 5271 268.378 5109 9911.822 10180.199
> 81920.0 81920.0 0.0 4097.0 655360.0 215879.9 7569408.0 7202846.9 43280.0 25855.8 5271 268.378 5110 9912.026 10180.404
> 81920.0 81920.0 0.0 4097.0 655360.0 215924.5 7569408.0 7202846.9 43280.0 25855.8 5271 268.378 5110 9912.026 10180.404
> 81920.0 81920.0 0.0 4097.0 655360.0 215932.7 7569408.0 7202846.9 43280.0 25855.8 5271 268.378 5110 9912.026 10180.404
> 81920.0 81920.0 0.0 4097.0 655360.0 216189.6 7569408.0 7202846.9 43280.0 25855.8 5271 268.378 5111 9912.347 10180.725
> 81920.0 81920.0 0.0 4097.0 655360.0 216208.6 7569408.0 7202846.9 43280.0 25855.8 5271 268.378 5112 9912.556 10180.934
> 81920.0 81920.0 0.0 4097.0 655360.0 216504.1 7569408.0 7202846.9 43280.0 25855.8 5271 268.378 5112 9912.556 10180.934
> 81920.0 81920.0 0.0 4097.0 655360.0 216517.9 7569408.0 7202846.9 43280.0 25855.8 5271 268.378 5112 9912.556 10180.934
> {quote}
> This continues even after we shut down our app. Nothing is connected to Cassandra any more, yet it is still stuck in full GC and cannot free up memory.
> Running nodetool tpstats shows that nothing is pending, all seems OK:
> {quote}
> Pool Name Active Pending Completed Blocked All time blocked
> ReadStage 0 0 69555935 0 0
> RequestResponseStage 0 0 0 0 0
> MutationStage 0 0 73123690 0 0
> ReadRepairStage 0 0 0 0 0
> ReplicateOnWriteStage 0 0 0 0 0
> GossipStage 0 0 0 0 0
> CacheCleanupExecutor 0 0 0 0 0
> MigrationStage 0 0 46 0 0
> MemoryMeter 0 0 1125 0 0
> FlushWriter 0 0 824 0 30
> ValidationExecutor 0 0 0 0 0
> InternalResponseStage 0 0 23 0 0
> AntiEntropyStage 0 0 0 0 0
> MemtablePostFlusher 0 0 1783 0 0
> MiscStage 0 0 0 0 0
> PendingRangeCalculator 0 0 1 0 0
> CompactionExecutor 0 0 74330 0 0
> commitlog_archiver 0 0 0 0 0
> HintedHandoff 0 0 0 0 0
> Message type Dropped
> RANGE_SLICE 0
> READ_REPAIR 0
> PAGED_RANGE 0
> BINARY 0
> READ 585
> MUTATION 75775
> _TRACE 0
> REQUEST_RESPONSE 0
> COUNTER_MUTATION 0
> {quote}
> We had this happen on 2 separate boxes, one with 2.0.6, the other with 2.0.8.
> Right now this is a total blocker for us. We are unable to process the customer data and have to abort in the middle of large processing.
> This is a new customer, so we did not have a chance to see if this occurred with 1.1 or 1.2 in the past (we moved to 2.0 recently).
> We have the Cassandra process still running, pls let us know if there is anything else we could run to give you more insight.
--
This message was sent by Atlassian JIRA
(v6.2#6252)