You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (JIRA)" <ji...@apache.org> on 2011/01/27 22:14:45 UTC
[jira] Created: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Read repair causes tremendous GC pressure
-----------------------------------------
Key: CASSANDRA-2069
URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.7.1
Reporter: Brandon Williams
To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Remaining Estimate: 16h
Original Estimate: 16h
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.3
>
> Attachments: 2069-v10.txt, 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069-v6.txt, 2069-v7.txt, 2069-v8.txt, 2069-v9.txt, 2069.txt
>
> Original Estimate: 16h
> Remaining Estimate: 16h
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Attachment: (was: 2069.txt)
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069-v6.txt, 2069-v7.txt, 2069-v8.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Fix Version/s: (was: 0.7.2)
0.7.3
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.3
>
> Attachments: 2069-v10.txt, 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069-v6.txt, 2069-v7.txt, 2069-v8.txt, 2069-v9.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Attachment: 2069-v10.txt
rebased v10
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v10.txt, 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069-v6.txt, 2069-v7.txt, 2069-v8.txt, 2069-v9.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Matthew F. Dennis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990416#comment-12990416 ]
Matthew F. Dennis commented on CASSANDRA-2069:
----------------------------------------------
I've observed similar issues with large a large newgen, largish memtables and writing at RF>1 at CL.Q/CL.ALL though it's not as immediate as it is with RR.
We should look into a way at bounding the number of outstanding requests (RR or "send to replica"), or more aggressively timing out the older requests once we hit some configurable threshold of outstanding requests.
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Fix For: 0.7.2
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990824#comment-12990824 ]
Brandon Williams commented on CASSANDRA-2069:
---------------------------------------------
No GC pressure now, however RR does not appear to actually occur.
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2069) Read repair causes
tremendous GC pressure
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990343#comment-12990343 ]
Brandon Williams edited comment on CASSANDRA-2069 at 2/3/11 10:37 PM:
----------------------------------------------------------------------
MAT indicates there are millions of pending futuretasks in StorageProxy.repairExecutor.
was (Author: brandon.williams):
MAT indicates there are millions of pending futuretasks in StorageService.repairExecutor.
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Fix For: 0.7.2
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990343#comment-12990343 ]
Brandon Williams commented on CASSANDRA-2069:
---------------------------------------------
MAT indicates there are millions of pending futuretasks in StorageService.repairExecutor.
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Fix For: 0.7.2
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988130#action_12988130 ]
Jonathan Ellis commented on CASSANDRA-2069:
-------------------------------------------
In the log snippets I saw the heap was legitimately nearly-full during the repair process. What does MAT show is using all the memory?
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Fix For: 0.7.2
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Attachment: 2069-v5.txt
v5 fixes a confusion of endpoints and handler.endpoints
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Attachment: 2069-v10.txt
v10 fixes a race where a quick repair would remove the data needed for the response to client. it also splits repair and digest-processing resolvers into different classes.
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v10.txt, 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069-v6.txt, 2069-v7.txt, 2069-v8.txt, 2069-v9.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Attachment: 2069-v6.txt
v6 adds the new repair callbacks to DES latency tracking.
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069-v6.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995020#comment-12995020 ]
Hudson commented on CASSANDRA-2069:
-----------------------------------
Integrated in Cassandra-0.7 #280 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/280/])
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v10.txt, 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069-v6.txt, 2069-v7.txt, 2069-v8.txt, 2069-v9.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Attachment: 2069-v4.txt
v4 fixes build
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Attachment: 2069-v7.txt
v7 rebases post-1530.
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069-v6.txt, 2069-v7.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987790#action_12987790 ]
Jonathan Ellis commented on CASSANDRA-2069:
-------------------------------------------
is this also true for 0.6?
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Fix For: 0.7.2
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990862#comment-12990862 ]
Jonathan Ellis commented on CASSANDRA-2069:
-------------------------------------------
what does debug log show?
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Attachment: (was: 2069-v10.txt)
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069-v6.txt, 2069-v7.txt, 2069-v8.txt, 2069-v9.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Attachment: 2069.txt
bq. RR does not appear to actually occur
No digest reads are being sent but I don't know why. v8 adds better debug logging around ReadCallback.endpoints creation which governs that.
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069-v6.txt, 2069-v7.txt, 2069-v8.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990433#comment-12990433 ]
Jonathan Ellis commented on CASSANDRA-2069:
-------------------------------------------
Working on another approach that should let us throw away repair handlers in the expected case that everyone responds quickly. This will be kinder to new gen gc since we won't have nearly as many survivors.
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990392#comment-12990392 ]
Brandon Williams commented on CASSANDRA-2069:
---------------------------------------------
Bisecting this indicates two problems. First, the tremendous GC pressure doesn't exhibit until we increase the newgen size. If that is ignored and we only observe whether stress.java moves forward (it has the 'keep trying' rather than 'keep going' behavior) then the culprit is r1053245.
Both cases show that RR requests can grow unbounded, which is probably ok, but if it's the same request over and over we still queue it as many times as it's requested.
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Fix For: 0.7.2
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Attachment: 2069-v2.txt
v2 adds a READ_REPAIR stage and does resolve of digests that were not checked for the client result on that stage as soon as response() collects all the replies. If there is a mismatch and we do a re-read of full resultset, we also check those results on the RR stage based on response() (in AsyncRepairRunner, now in ReadCallback.)
I preserved the feature from v1 of not doing repairs if the RR stage is full.
Most of the code changes are about getting the right information into ReadCallback (e.g. endpoints) and some ceremony to make static typing happy (IReadCallback).
A side benefit is that StorageProxy.fetchRows is significantly cleaner (no more commandEndpoints or repairs collections).
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v2.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987801#action_12987801 ]
Brandon Williams commented on CASSANDRA-2069:
---------------------------------------------
No, only 0.7
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Fix For: 0.7.2
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990865#comment-12990865 ]
Brandon Williams commented on CASSANDRA-2069:
---------------------------------------------
Nothing interesting, afaict. Just a lot of this:
{noformat}
DEBUG 23:19:42,094 get_slice
DEBUG 23:19:42,094 get_slice
DEBUG 23:19:42,240 ReadCallback blocking for 1 responses
DEBUG 23:19:42,240 ReadCallback blocking for 1 responses
DEBUG 23:19:42,093 ReadCallback blocking for 1 responses
DEBUG 23:19:42,092 reading data for SliceFromReadCommand(table='Keyspace1', key='30343939323034', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5) locally
DEBUG 23:19:42,091 Read: 21 ms.
DEBUG 23:19:42,091 LocalReadRunnable reading SliceFromReadCommand(table='Keyspace1', key='30353032333935', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5)
DEBUG 23:19:42,091 Read: 17 ms.
DEBUG 23:19:42,241 Read: 159 ms.
DEBUG 23:19:42,091 Read: 12 ms.
DEBUG 23:19:42,091 reading data for SliceFromReadCommand(table='Keyspace1', key='30333636313035', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5) locally
DEBUG 23:19:42,091 LocalReadRunnable reading SliceFromReadCommand(table='Keyspace1', key='30343831383238', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5)
DEBUG 23:19:42,091 get_slice
DEBUG 23:19:42,242 Read: 161 ms.
DEBUG 23:19:42,242 ReadCallback blocking for 1 responses
DEBUG 23:19:42,091 get_slice
DEBUG 23:19:42,090 Read: 17 ms.
DEBUG 23:19:42,243 ReadCallback blocking for 1 responses
DEBUG 23:19:42,090 Read: 19 ms.
DEBUG 23:19:42,090 reading data for SliceFromReadCommand(table='Keyspace1', key='30353131363631', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5) locally
DEBUG 23:19:42,089 ReadCallback blocking for 1 responses
DEBUG 23:19:42,243 reading data for SliceFromReadCommand(table='Keyspace1', key='30353532343833', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5) locally
DEBUG 23:19:42,244 LocalReadRunnable reading SliceFromReadCommand(table='Keyspace1', key='30353532343833', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5)
DEBUG 23:19:42,244 Read: 1 ms.
DEBUG 23:19:42,243 reading data for SliceFromReadCommand(table='Keyspace1', key='30363137363033', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5) locally
DEBUG 23:19:42,242 reading data for SliceFromReadCommand(table='Keyspace1', key='30363234303332', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5) locally
DEBUG 23:19:42,242 LocalReadRunnable reading SliceFromReadCommand(table='Keyspace1', key='30333636313035', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5)
DEBUG 23:19:42,241 LocalReadRunnable reading SliceFromReadCommand(table='Keyspace1', key='30343939323034', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5)
DEBUG 23:19:42,245 Read: 3 ms.
{noformat}
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Attachment: 2069-v8.txt
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069-v6.txt, 2069-v7.txt, 2069-v8.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Attachment: 2069.txt
Attached patch does two things:
- allows the repair executor to use threads = # of cores
- stops sending repair requests when the executor queue gets too large. ("dropped" requests will be logged by MessagingService along with the other overload-scenario dropping.)
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994569#comment-12994569 ]
Brandon Williams commented on CASSANDRA-2069:
---------------------------------------------
+1, RR works, GC pressure is gone.
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v10.txt, 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069-v6.txt, 2069-v7.txt, 2069-v8.txt, 2069-v9.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Attachment: 2069-v3.txt
bq. I preserved the feature from v1 of not doing repairs if the RR stage is full.
Removed this for v3. It's complexity we don't need to solve a non-problem, when we're not hanging on to each callback until RPC_TIMEOUT (as demonstrated by 0.6).
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v2.txt, 2069-v3.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Attachment: 2069-v9.txt
v9 fixes ReadCallback construction and adds more asserts around AsyncRepairRunner.
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 0.7.2
>
> Attachments: 2069-v2.txt, 2069-v3.txt, 2069-v4.txt, 2069-v5.txt, 2069-v6.txt, 2069-v7.txt, 2069-v8.txt, 2069-v9.txt, 2069.txt
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2069) Read repair causes tremendous GC
pressure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2069:
--------------------------------------
Fix Version/s: 0.7.2
> Read repair causes tremendous GC pressure
> -----------------------------------------
>
> Key: CASSANDRA-2069
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2069
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Brandon Williams
> Fix For: 0.7.2
>
>
> To reproduce: start a three node cluster, insert 1M rows with stress.java and rf=2. Take one down, delete its data, then bring it back up and issue 1M reads against it. After the run is done you will see at least 1 STW long enough to mark the node as dead, often 4 or 5.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.